The Man-Machine Integration Design & Analysis System (MIDAS):

Recent Improvements

Sandra G. Hart

Brian F. Gore

Peter A. Jarvis

NASA Ames Research Center

Moffett Field, CA 94035

Sandra.G.Hart@NASA.gov/650 604 6072

10/19/04

SGH 10/04-2

Outline

Human Performance Modeling

MIDAS Phase 1: Initial design

Early applications

MIDAS Phase 2: Move from Lisp to C++

Recent applications

MIDAS Phase 3: PC Port/Integrate Apex

SGH 10/04-3

Human Performance Models: Components

Psychological Models

ProceduralModels

Vehicle Models

Equipment Models

Sensory Models

Anthropometric Models

Environment Models

Team/Org Models

Biodynamic Models

Timeline

Task Network

Performance: WL

Performance: Time

Performance: SA

Performance: Errors

Visualization

FoV/Reach Envelope

Model Architecture, Library, Tools

SGH 10/04-4

Human Performance Models: Architectures

Task network: Top-down, based on sequences of human/system tasks (derived from task analysis

MicroSaint WinCrew Crewcut

IPME IMPRINT

ACT-R MIDAS/ AirMIDAS D-OMAR

Soar APEX SAMPLE

Cognitive: Bottom-up, combine theory-based models of memory, decision making, perception, attention, movement, etc

Vision: Computational representation of the way the human visual system processes an image to predict performance given image characteristics

ORACLE NASA Standard Visual Observer

NASA Text Visibility

Optimetrics Visual Perf Model

Georgia Tech Vis Model

Anthropometric/Biodynamic: Physical characteristics of human body; static & dynamic; population characteristics; limitations [RAMSIS, JACK]

Psychological theories, mathematical models, descriptive functions

SGH 10/04-5

Human Performance Models can…

Generate hardware, software, training requirements for tasks that will involve human operators

Depict operators performing tasks in prototype workspaces and/or in remote or risky environments

Perform tradeoff analyses among alternative designs and candidate procedures, saving time and money

Identify general human/system vulnerabilities to estimate overall system performance and reliability

Provide dynamic, animated examples for training and developers

Generate realistic schedules and procedures

Phase 1

SGH 10/04-7

OverviewA comprehensive suite of computational tools - - 3D rapid prototyping, models of perception, cognition, response, real- and fast-time simulation, performance analysis, visualization - - for designing and analyzing human/machine systems was developed primarily in Lisp on a fleet of SGIs

Run Time Visualization

Data AnalysisData Input

SGH 10/04-8

Features Pioneered the development of an engineering design

environment with integrated tools for rapid prototyping, visualization, simulation and analysis

Advanced the capabilities and use of computational representations of human performance in design including a state of the art anthropometric model (Jack®)

Flexible enough to support a range of potential users and target applications

But…. Component models written in Lisp, Fortran, C, C++ Required a suite of SGI machines Modeled a single operator Time based rather than event based; scheduler established

optimal inter-leaving of task components No emergent behaviors

SGH 10/04-9

Richmond, CA Police: 911 Dispatch

Goal: Upgrade the facilities and procedures used in the 911 dispatch facility

Accomplished: Modeled control console

and dispatch activities in MIDAS

Evaluated prototype graphical decision aid

SGH 10/04-10

US Army Air WarriorGoal: Establish baseline performance measures for crews flying Longbow Apache with and without MOPP gear

Accomplished: Modeled copilot/gunner with Jack® (95th male <> 5th female) Rendered cockpit using CAD files from manufacturer Simulated performance of more than 400 activities Measured reach, FoV, workload, timelines

SGH 10/04-11

Short Haul/Civil Tiltrotor

Goal: Evaluate crew performance/workload issues for steep (9º), noise abatement approaches into a vertiport

Accomplished: Constructed MIDAS

models of normal and aborted approaches

Contrasted impact of manual vs automated nacelle control modes

SGH 10/04-12

NASA Shuttle Upgrade

Goal: Support development of an advanced orbiter cockpit with an improved display/control design

Accomplished: Created virtual rendition of current shuttle cockpit Conducted simulation of first 8 min of nominal ascent Provided quantitative measures of workload/SA, timing

Phase 2

SGH 10/04-14

Features

Decreased model development from months to weeks Increased run-time efficiency from 50x RT to near RT Multiple operators Modeled external vision, audition, situation awareness Conditional behaviors emerging from interaction of

top-down goals and environmentally driven contexts Option of non-proprietary “head & hands” model

But… The interface still user un-friendly SGI platform Cognitive models no longer state of the art Performance moderating functions not integrated

SGH 10/04-15

Overview of Architecture

SGH 10/04-16

User Interface The interactive graphical user interface is used to create

models, specify and run simulations, and view data. It is organized into a hierarchical series of screens or editors that are navigated with tabs

Different views of the simulation are offered: Structure, Geometry, Outline, Animation, Real-time/post-hoc data

SGH 10/04-17

Vehicle Models A modeled vehicle represents the combination of a guidance/

dynamics model and a visual representation The guidance/dynamics model moves the vehicle along a

prescribed route. MIDAS provides two:NoE helicopter modelSimple point mass model

(used to model arbitrary vehicles in a generic way)

The visual representation is CAD geometry chosen from the MIDAS library or developed by the modeler.

SGH 10/04-18

Environment Model Tools are provided to model the environment outside the crew

station (e.g., terrain, weather, etc) Terrain is modeled as a single object Features are simple objects that have no inherent behavior and do

not move One weather condition may be applied to the environment by

specifying lighting/visibility (these are used by the visual perception model)

SGH 10/04-19

Crew Station/Equipment Models The “crew station” is a collection of equipment with which operators interact Crew station models may

be given a graphical representation for animation

Multiple crew stations per vehicle and multiple operators per crew station possible

QuickTime™ and aVideo decompressor

are needed to see this picture.

SGH 10/04-20

Anthropometric Models

Anthropometric models provide an animated, 3D graphical representation of one or more modeled human operators for visualization

Jack ® (developed at U Penn/distributed by UGS): full-body figure & realistic movements

Head and Hands model: government-developed representation adequate for many purposes for users without a Jack license

SGH 10/04-21

Vision ModelsVisual attention modeled as single “cone”, varying from 3-15º based on task type.

External vision:Peripheral: 160 degreesFoveal: 2.5 degrees

Perceivability: f(visibility, size, distance, local contrast ratio)Perception level: f(dwell time, perceivability)Levels of Perception:

DetectionRecognitionIdentification

Internal vision: Symbolic (check read)Digital (exact value)Text (character string)

SGH 10/04-22

Auditory Model

Only within crew stationExternal sounds are represented only if channeled through equipment

Two Stages of Processing:DetectionComprehension

Content: Verbal stringsSignals

All or none processing (Interruptions disrupt entire message)

SGH 10/04-23

Symbolic Operator Models Significant advance over earlier version, which required

specification of all activities at primitive task level High-level scripting language, Operator Procedure Language

(OPL) serves as front-end to a reactive planning system (RAPS) User-supplied procedures are instructions for accomplishing tasks Manages knowledge and beliefs, integrates human actions with

scenario events

SGH 10/04-24

Simpler model than in MIDAS 1.0 Distinction between long-term/short-term memory was lost Memories are represented as a database of assertions or beliefs

that are symbolic expressions describing the property of objects

Memory can be examined by powerful tools in a querying

language built into OPL

Memory Model

SGH 10/04-25

Attention Model Based on Wicken’s Multiple Resource Model. Acts as a mediator that maintains an account of attention

resources in six different “channels” Necessary attention resources must be available before primitive

tasks are initiated Task onset may be delayed if insufficient resources

MANUAL

VOCAL

ENCODING RESPONDING

CENTRALPROCESSING

VISUAL

AUDITORY

SPATIAL

VERBAL

SPATIAL

VERBAL

CODES

MO

DA

LIT

IES

STAGES

RESPONSES

SGH 10/04-26

Output Behavior Models

If required resources are available an activity that corresponds to a primitive procedure is created

Physical actions and their effects on equipment or environmental objects are modeled, regulated by a motor control process

60+ primitive tasks are available in a Procedure Library with pre-defined load values; easy to add more

Visual Auditory CognitiveSpatial

Cognitive Verbal

Manual Vocal

Estimate Time 0 0 0 2.0 0 0

Visual monitor 5.4 0 6.8 0 0 0

Type (1 hand) 5.9 0 0 5.3 7.0 0

Say message 0 0 0 5.3 1.0 4.5

Move object 5.0 0 1.0 0 2.6 0

SGH 10/04-27

Simulation System Engine/executive (uses discrete-event, fixed-time increment

approach for advancing the simulation) Data collection mechanisms for generating runtime data that is

graphically displayed which the simulation runs and is saved for post-run analyses

Event generation mechanism provides a way for timed events to occur on schedule or with stochastic variance

Provisioning system allows users to change the simulation and re-run without re-loading/re-starting

SGH 10/04-28

Workload & Situation AwarenessWorkload calculations based on McCracken & Aldrich (1988)

Load levels for Visual, Auditory, Cognitive, and Psychomotor dimensions are defined for task primitives on a scale of 1-7

Momentary load based on aggregation

SA calculations based on:Ratio of operator’s relevant knowledge/required knowledgeDistinguishes actual SA from perceived SA

Situational elements can be objects in the crew station or the environment that define a “situation” or are in the operator’s memory and are operationally relevant.

WL and SA values offer a powerful way to simulate realistic errors

QuickTime™ and aVideo decompressor

are needed to see this picture.

SGH 10/04-29

Validation: Search & Rescue Mission

QuickTime™ and aMicrosoft Video 1 decompressorare needed to see this picture.

SGH 10/04-30

Comparison of Models to Simulator Data

Nominal baseline approach/landing and late runway reassignment (sidestep) with and without SVS display

850' Ceiling

1000' Lineup on Final

Parallel Runways

650' Missed Approach

ATC-Commanded Runway Sidestep

Preliminary timeline, SA, attention, wkld, analysis,task execution times error vulnerabilities

ACT-R/PM

U of Illinois

Rice University

Air MIDAS

San Jose State University

A - SA

U of Illinois

IMPRINT/ACT-R

MAAD & Carnegie Mellon

D-OMAR

BBN Technologies

MIDASNASA-ARC/Army

Goodness-of-fit of individual model outputs to empirical data

SGH 10/04-31

Nominal Approach & Landing Simulation PF scanning for TFX, runway PNF monitoring PFD, Nav PF/PNF monitoring radio Flaps 30º/set & confirm PF requests before landing

checklist PNF checks/responds hear

down PF confirms visually/verbally PNF checks/responds flaps 30 PF confirms visually/verbally PNF checks/responds speed

brakes set PF confirms visually/verbally PNF declares checklist

complete PF sets/declares DA at 650 PNF visually confirms DA set Note passing FAF Confirms final descent initiated

QuickTime™ and aVideo decompressor

are needed to see this picture.

SGH 10/04-32

Traffic Call During Approach Final approach checklist

is complete ATC call with traffic

advisory Both pilots scan for

traffic “I don’t see it” Neither pilot notices as

the decision altitude is passed

After the fact, the First Officer notices: “We’re past FAF and not descending”

Crew must decide whether to continue with the approach or abort

QuickTime™ and aVideo decompressor

are needed to see this picture.

SGH 10/04-33

Life Sciences Glove BoxVirtual Glovebox

Onboard KC-135

Life Sciences Glovebox Payload Development Unit received at Ames from the National Aerospace

Development Agency of Japan (NASDA)

MIDAS rendering

Challenges: Astronauts must follow detailed instructions within

strict time constraints; failure to do so introduces risk of science mission failure

Role of Computational ModelingPredict interactive influences of microgravity

(posture, bracing, precise movements, placing, moving, stowing) to develop/evaluate procedures

Watching an animated dry run enables efficient communication among scientists, implementers, astronauts; more effective training

SGH 10/04-34

Life Sciences Glove Box Simulation

Goal: Predict astronauts’ performance of complex experiments designed to answer questions about living organisms’ adaptation to the space environment

Objectives: Evaluate feasibility of following proposed procedures within time/performance constraints; ID factors that will increase risk of mission failure [e.g., waiting too long to photograph slides; interruptions; task requires (unavailable) resource(s)]

The Task: Turn on experimental equipment (monitor, microscope, camera) Measure cell density/viability for each of 6 samples

• Invert sample vial• Place aliquot of sample on slide• Place drop of viability stain in sample• Record time on sample record• Place cover slip on slide• Observe on microscope• Take photographs within specific time window

Dispose of trash, return vials to containers, turn equipment off

SGH 10/04-35

Cell Staining/Photographing Experiment

QuickTime™ and aVideo decompressor

are needed to see this picture.

Phase 3

SGH 10/04-37

MIDAS v3.0 Features

Runs on high-end PC Simple model of microgravity influence on performance Physics model of microgravity impact on objects available Simple within-task fatigue model implemented Fatigue state model (U Penn/Astronaut Scheduling Assistant)

selected Notion of task duration - - how long a task should take as well as

how long it did take Grasping, moving, manipulating objects in workspace Apex will become the heart of the Task Manager and enable multi-

tasking, task prioritization, shedding, deferral, resumption Task primitive definitions include failure modes (time/quality) that

enable the occurrence of emergent behaviors Mission success/performance measures computed: vulnerability

to error, slipped schedules; performance degradation

SGH 10/04-38

MIDAS v3.0 Structure

Workstation Geometry

List of Tasks/Procedures

Mission Environment

Operator Characteristics

Physical SimulationPerceivesAttends

Moves/ActsChanges

Cognitive simulation Behavior modifiers

Situation AwarenessError, Workload

Timeliness

Commands Results

Task Network

Dynamic Animation

Mission success

Task stateOperator state

Task executions

Dynamic models

Timeline

Performance measures

Fit/Reach/Vis envelope

LibraryPrimitive tasksHuman model

Task ManagerPlans

MonitorsRemembers

SensesActuates

SGH 10/04-39

Typical Outputs

SGH 10/04-40

QuickTime™ and aAnimation decompressor

are needed to see this picture.

“Fresh”

SGH 10/04-41

“Tired”

QuickTime™ and aAnimation decompressor

are needed to see this picture.

SGH 10/04-42

PC Version: Early Simulation

QuickTime™ and aMicrosoft Video 1 decompressorare needed to see this picture.

SGH 10/04-43

Conclusion MIDAS 3.0 now operates on a PC platform and will soon incorporate

significantly enhanced cognitive model (Apex)

MIDAS 3.0 gives users the ability to model the functional and physical aspects of a variety of operators, systems, and environments.

It brings these models together in an interactive, event-filled simulation for quantitative and visual analysis

The interplay between top-down and bottom-up processes and a suite of performance modifying functions enables the emergence of un-forseen, un-scripted behaviors

The government has done what it set out to do - - spur development of human performance modeling tools integrated into a design environment

Our goal is to continue to add functionality with each new application