16.842 Fundamentals of Systems Engineeringdspace.mit.edu/bitstream/handle/1721.1/103819/16...Preliminary Design System/software Reqs. Unit Development User survey, needs analysis,

16.842

Fundamentals of Systems Engineering

Human-Systems Engiineeriing

V‐Model – Oct 23, 2009

Systems EngineeringStakeholder Lifecycle y g gOverviewAnalysis

Requirements Commissioning

yManagement

Definition

System Architecture Verification and

gOperationsCost and Schedule

Management

Concept Generation

Tradespace ExplorationC t S l ti

System IntegrationI t f M t

Validation

Concept Selection

Design DefinitionM l idi i li O i i i

Interface ManagementHuman Factors System

SafetyMultidisciplinary Optimization

y

u co s de o s o e e ow o

16.842

Traditional Systems Engineering Process Model*

ACQUISITION PHASE UTILIZATION PHASE

N E E D

Conceptual-Preliminary Design

Detail Design and Development

Production and/or Construction

Product Use, PhaseouDisposal

t, and

• Operational requirements drive technical f hi h d i h f tperformance measures which drive human factors

requirements….. – Human considerations often are low ppriorityy

*Blanchard, B. S., & Fabrycky, W. J. (1998). Systems Engineering and Analysis (3rd ed.). Upper Saddle River, NJ: Prentice Hall.

Or stuck out on the periphery…

Systems EngineeringStakeholder Lifecycle

16.842

y g gOverviewAnalysis

Requirements Commissioning

yManagement

Definition

System Architecture Verification and

gOperationsCost and Schedule

Management

Concept Generation

Tradespace ExplorationC t S l ti

System IntegrationI t f M t

Validation

Concept Selection

Design DefinitionM l idi i li O i i i

Interface ManagementHuman Factors System

SafetyMultidisciplinary Optimization

y

Results of “Classic” SE Methods 16.842

These images have been removed due to copyright restrictions.

16.842

Three Mile Island

• March 28th, 1979 • Main feedwater pump failure, caused reactor to shut

down • R li f Relief vallve openedd to redduce pressure bbut bbecame

stuck in the open position – No indication to controllersNo indication to controllers – Valve failure led to a loss of reactant coolant water

• No instrument showed the coolant level in the reactor• Operators thought relief valve closed & water level too

high – i hHigh stress – Overrode emergency relief pump

Three Mile Island, cont. 16 842 16.842

• System worked as designed, automation worked correctly – Confirmation bias: people seek out information to confirm

a prior belief and discount information that does not support this beliefsupport this belief

– Operators selectively filtered out data from other gauges to support their hypothesis that coolant level was too high

This image has been removed due to copyright restrictions.

16.842

The Spiral Systems Engineering Process Model*

*Boehm, B. (1988). A Spiral Model of Software Development and Enhancement. Computer, 61-72.

Risk analysis

Risk analysis

Risk analysis

Riskanal

RQTS planlife cycle

planConcept ofoperation

Emulations

SoftwareRQTS

Requirementsvalidation

Developmentplan

Proto-type1

Prototype2Prototype3

Operationalprototype

BenchmarksModels

Detaileddesign

Code

Unit testDesign validationand verification

Integrationandtest

Develop, verify nextlevel product

Plan next phases

Evaluate alternativesidentify, resolve risks

Determine objectives,alternatives, constraints

Commitment,partition

Progress through steps

Acceptancetest

Implemen-tation

Integrationand test plan

Softwareproductdesign

Review

Cumulative cost

Image by MIT OpenCourseWare.

Human Syystems Enggineeringg* 16.842

* Aptima, Inc. rendition*

System/Software Acquisition Cycle

"Vision"

Installation

Integration & Test Detailed Design

Preliminary Design

System/software Reqs.

Unit Development

User survey, needs analysis, etc.Artifact and like-system evaluation

Finalize customer supportConduct on-site assessment

Provide training materials

Plan performance testsScenario-based assessment

Develop training materials

Design tradeoff and workflow analysis

Needs and task analysis

Feasibility assessmentPerformance and usability reqs.

HE labor/budget planning

Storyboards and demonstrationsUI design standards

Detailed UI designs and prototypesOn-line help and documentation

Innovative concepts for nextversion release

HE heuristic review

Define performance and effectiveness criteria

Usability evaluation of prototypes


A Simplified HSE approachp ppPlanning

16.842

Mission & Scenario Analysis AnalysisAnalysis

Function Analysis

Function Allocation

Task Analysis

Design Test & Evaluation

System Design

–

Mission & Scenario Analysis 16.842

• Stakeholder analysis • Do users always know what they

want/need?– Revolutionary vs. evolutionary systems

• Interviews & observations • Work process flows•• This is a critical step and should beThis is a critical step and should be

agreed upon before moving forwardThe most ill defined but the most important step The most ill-defined but the most important step

A Simplified HSE approachp ppPlanning

16.842

Mission & Scenario Analysis AnalysisAnalysis

Function Analysis

Function Allocation

Task Analysis


System Design

16.842

Determining function allocation

• More art than science • Steps:

– Identifyy functions • Use Cases • Interviews • Customer requirements

– Identify who is best suited for each function• Human or automation or shared? • Static vs. dynamic/adaptive

• Sounds easy!

16.842 Function Allocation via Fitts’ List?

Attribute Machine Human Speed Superior Comparatively slow

Power Output

Superior in level in consistency Comparatively weak

ConsistencyConsistency Ideal for consistent, repetitive actionp Unreliable, learning & fatigue ag g factor

Information Capacityp y

Multi-channel Primarily single channel

Memory Ideal for literal reproduction, access restricted and formal

Better for principles & strategies, access versatile & innovative

ReasoningReasoning Computation

Deductive, tedious to program, fast, p g , & accurate, poor error correction

Inductive, easier to program, slow,, p g , , accurate, good error correction

Sensing Good at quantitative assessment, poor at pattern recognitionp p g

Wide ranges, multi-function, judgmentj g

Perceiving Copes with variation poorly, susceptible to noise

Copes with variation better, susceptible to noise

Hollnagel, 2000

To automate or not to automate? 16.842

Function Allocation Criteria16 842

1: No difference in the relative capabilities of human & machine. el

lent

16.842

2: Human performance > machine performance.

3: Machine performance > human.3

Exc

4: Machine performance is so poor that the functions should be allocated to humans.

5 H f i

15

Mac

hine

5: Human performance is so poor that the functions should be allocated to machine.

6: Unacceptable performance by

2

M

p p yboth human and machine.

Three function allocation criteria:Balance of value

46

Uns

atis

fact

ory

• Balance of value• Utilitarian & cost-based

allocation• Allocation for affective or

iti t

Unsatisfactory ExcellentHuman

Price, 1985

16.842

Sheridan and Verplank’s 10 Levels of Automation of Decision and Action SelectionAutomation of Decision and Action Selection

Automation Level

Automation Description

1 The computer offers no assistance: human must take all decision and actions.

2 The computer offers a complete set of decision/action alternatives, or

3 narrows the selection down to a few, or

4 suggests one alternative, andgg ,

5 executes that suggestion if the human approves, or

6 allows the human a restricted time to veto before automatic execution, or

77 t t ti ll th il i f h dexecutes automatically, then necessarily informs humans, and

8 informs the human only if asked, or

9 informs the human only if it, the computer, decides to.

10 The computer decides everything and acts autonomously, ignoring the human.

The Human-Automation Paradox 16.842

These images have been removed due to copyright restrictions.

Adaptive Automation 16.842

• Dyynamic function allocation

• Mode This image of a crashed jet has been removed due to

Confusion copyright restrictions. Confusion – A problem of

intent• Mixed initiative

•• FlexibleFlexibleautomation

Functional Requirements

Planning

16.842

Mission & Scenario Analysis Analysis

Function Analysis

y

Function Allocation

Task Analysis

DesignFunctional

Requirements

Test & Evaluation

System Design

q

Task Analysis

Planning

16.842


Function Analysis

y

Function Allocation

Task Analysis


System Design

–

16.842

Task Analysis

• Taylor • D tDetermiiniing wh that an operattor mustt accomplili sh t h to

meet a mission goal –Interactions both on a local & s stem le el are critical Interactions both on a local & system level are critical – Will contain actions and/or cognitive processes

• Flow process charts Flow process charts, operational sequence• operational sequence diagrams, critical task analysis

Attempt to understand how a particular task couldAttempt to understand how a particular task could exceed human limitations, both physical and cognitive

• Cognitive task analysisCognitive task analysis – Shift from system control to systems management.

•

16.842

Cognitive Task Analysis (CTA)

• Goal: To analyze and represent the knowledge and cognitive activities needed in complex work domains cognitive activities needed in complex work domains

• CTA is generally a descriptive modeling technique of workers’ knowledgge and coggnition – As opposed to Computational Cognitive Models (CCM) – Knowledge Elicitation is a central feature

• Experts vs Novices Experts vs. Novices

• Evolutionary systems vs. revolutionary systems • Backgground Research

– Standards, procedures, manuals, organizational charts

• Field Studies – I b h l i d hi h fid li i lIn both real environments and high fidelity simulatiions

• Questionnaires/Surveys

16.842

CTA, Cont. 16 842

• Interviews – Individuals vs. focus ggroupps – Critical Incident Technique/Critical Decision Method

• Observations – Verbal protocols

• Design Reviews – Usability, Expert, Heuristic

• Problems with CTA – Labor intensive – Generate much data that is difficult to analyze – Gap between CTA and design – Opportunistic

16.842

SRK* Taxonomy of Cognitive Control

• Skill-Based Behavior (SBB) – Motor control/automaticity in perception-action cycle

• Rule-Based Behavior (RBB) – Procedural, stored if-then-else rules that dictate action

• Knowledge-Based Behavior (KBB) – Serial, analytical reasoning based on a mental model

(internal, symbolic representation of environmental constraints and relationships in the environment )constraints and relationships in the environment.)

• Which of these should be automated?•• Analyst’s best guess for SRK assignment?Analyst’s best guess for SRK assignment?

(*Rasmussen, 1976)

The Need for Iteration

Planning

16.842

Mission & Scenario Analysis

Function Analysis

Function Allocation

Functional Requirements?

Task Analysis

Test & Evaluation

System Design

Risk analysis

Risk analysis

Risk analysis

Riskanal

RQTS planlife cycle


Emulations

SoftwareRQTS


Developmentplan

Proto-type1



BenchmarksModels

Detaileddesign

Code


Integrationandtest


Plan next phases





Acceptancetest

Implemen-tation



Review

Cumulative cost


Information Requirements

Planning

16.842


Function Analysis

y

Function Allocation

Task Analysis

DesignInformation

Requirements

Test & Evaluation

System Design

Requirements

Prototyping

Planning

16.842


Function Analysis

y

Function Allocation

Task Analysis


System Design

16.842

The Spiral Systems Engineering Process Model*


Risk analysis

Risk analysis

Risk analysis

Riskanal

RQTS planlife cycle


Emulations

SoftwareRQTS


Developmentplan

Proto-type1



BenchmarksModels

Detaileddesign

Code


Integrationandtest


Plan next phases





Acceptancetest

Implemen-tation



Review

Cumulative cost


–

16.842

Prototyping & HSE

• Does design meet functional requirements? Will lead to design requirements Willlead to design requirements

– Elegant usability is not the primary focus • Low fidelity vs. medium/high fidelityLowfidelity vs. medium/high fidelity • Feedback & interactivity

– Particippatoryy desi ggn – Wizard of Oz

• Breadth vs. depth – Front end vs. back end / Horizontal vs. vertical

• DANGER – Research vs. product – Decision support design – not cool interface design

Prototyping Fidelity

Hybrid

16.842

LowHigh

Operational

Storyboarding

Operational Simulators

(Flight Decks, C2

Visual Basic

Paper Power Point

Commercial Gaming (MFS)

Java( g ,

Center(MFS)

Participatory Design Human/System Performance Testing

T & E

Planning

16.842


Function Analysis

y

Function Allocation

Task Analysis


System Design

Test & Evaluation 16 842 16.842

• Concurrent testing • HumanHuman vsvs. system performance testingsystem performance testing

– Simulation – Demonstrations

• Usability evaluations – Subjective vs. objective

• Lab testing vs. field testing • Cost-benefit analysis

– Human trials are expensive! – Testing in the spiral stages

• Formal experimental design (16.470)• COUHES

The Sppiral Syystems EnggineeringgProcess Model*

16.842


Risk analysis

Risk analysis

Risk analysis

Riskanal

RQTS planlife cycle


Emulations

SoftwareRQTS


Developmentplan

Proto-type1



BenchmarksModels

Detaileddesign

Code


Integrationandtest


Plan next phases





Acceptancetest

Implemen-tation



Review

Cumulative cost


Resources 16.842

• Schraaggen, J.M., Chippman, S.F., & Shalin, V.L. (Eds) (2000). Cognitive task analysis. Mahwah, New Jersey: Lawrence Erlbaum Associates.

• ONR/Aptima Cognitive Task Analysis website • A Survey of Cognitive Engineering Methods and

Uses

MIT OpenCourseWarehttp://ocw.mit.edu

16.842 Fundamentals of Systems EngineeringFall 2009

For information about citing these materials or our Terms of Use, visit: http://ocw.mit.edu/terms.

http://ocw.mit.edu

http://ocw.mit.edu/terms

Documents

16.842 Fundamentals of Systems Engineeringdspace.mit.edu/bitstream/handle/1721.1/103819/16...Preliminary Design System/software Reqs. Unit Development User survey, needs analysis,