1

JUP Conference, 10-11 Jan 2002Hosted by Princeton University

Quantitative Experimental Results:

Automation to Support Time-Critical Replanning Decisions

Kip Johnson, MIT Dept. of Aero/Astro EngAdvisor Jim Kuchar, along with Dr. Charles Oman

Sponsored by Office of Naval Research

Disclaimer: The views expressed in this presentation are those of the authors and do not reflect the official policy or position of the U.S. Air Force, Department of Defense, or the U.S. Government.

2

Research Context

Time-Critical Decision-Makingie. Combat flight route planning

• Aviation, medicine, chemical and energy production, finances

Complex Problem• Unstructured aspects• Multiple competing interests and goals• Time pressures

Automation Integration• Cannot “see” everything (sensor limitations)• Human may not understand basis for automated

decisions

3

Experimental Goal

Discover the relationship between time pressures, automation assistance, and the resulting decision performance, both quantitatively and subjectively.

DecisionMaking

TimePressure

AutomationAssistance

ProblemComplexity

Human

Solution QualityRating

Quantitative&

SubjectiveScore

4

Replanning Task Description

InformationIntegration

Human

Auto

Solution

Hazards

Time toTarget

FuelConstraint

Environment:1st Order Effects

5

Experimental Protocol

4. Subject Modifies Flight Route Under Time Pressure

Minimize Threat Exposure and Time to Target Deviation

Meet Time to Target and Fuel Constraints

1. View Preplanned Mission

Finish

Start

Target

Rendezvous

Hazards

2. Change in Environment

•Hazard, Time to Target, or Fuel Update

3. Route Suggested with Varying Automation Assistance (BLUE)

New Hazard!

6

Independent Variables

Automation CategoriesNone: No Automation

original route remains

Time/Fuel: Constraint Information Only meets time to target & fuel constraints

Hazard: Hazard Information Only avoids/minimizes hazard exposure

Full: Integration of Constraint + Hazard Information minimizes hazard exposure + meets constraints

7

Independent Variables (2)

Time Pressure• 20, 28, 40, 55 seconds (logarithmic)

– capture performance change

• Unlimited time to find individual’s optimal performance

8

Dependent Variables

1. Quantitative human performance measured by route cost at end of time pressure.

Cost = Hazard Exposure (linear) + Time to Target Deviation (exponential)

Fuel = constraint

2. Subjective evaluations.

#

1 11 0

ln * * exp * 1colors

Route RouteSegment Color

tCost A Length Cost B a b

t

9

• 14 subjects: students, ave age = 25, 3 pilots, 3 females

• 3 hrs: 2 hr training, 1 hr data collection

Test Matrix:

• Greco-Latin Square Design• 4 base maps (M), each rotated for 16 effective

scenarios

Test Conditions

Time Pressure

AutoCategory

20 28 40 55 None M1 M2 M3 M4

ToT/Fuel M2 M1 M4 M3 Hazard M3 M4 M1 M2

Full M4 M3 M2 M1

10

0.0942

0.0307

-0.2526

0.1277

-0.30

-0.25

-0.20

-0.15

-0.10

-0.05

0.00

0.05

0.10

0.15

0.20

None ToT & Fuel Hazard Full

Automation Category

Res

idu

al S

core

Repeated Measures ANOVA

Full auto assistance is sig. best

Hazard auto assistance is sig. better than none

Automation Effects

Quantitative Analysis

IncreasingPerformance

11

0.0653

-0.0222

-0.0030

-0.0400

-0.08

-0.06

-0.04

-0.02

0.00

0.02

0.04

0.06

0.08

0.10

20 28 40 55Time (sec)

Res

idu

al S

core

• Repeated Measures

ANOVA

• No sig. performance

improvement after

28 sec

Quantitative Analysis (2)

Time Pressure Effects

IncreasingPerformance

12

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

15 25 35 45 55Time Pressure (sec)

Ab

solu

te S

core

NoneToT/FuelHazardFull

Quantitative Analysis (3)

• Only none had sig. improvement with time• Hazard better than none and time/fuel < 55 sec, sig. at 40 sec• None outperforms hazard and time/fuel at 55 sec, while is the

worst at 20 sec• Performance decreases at times dependent on auto level

InteractionEffects

optimalscore

IncreasingPerformance

13

• Idea of “Characteristic Time” (CT)– Period of beneficial

time from having auto assistance

• Possible linear metric– Similar slopes

– Intercept quantifies relative auto benefit

• CTFull > 55 sec

• CTHazard 55 sec

• CTTime/Fuel 20 sec

Temporal Benefit of Automation

y = -0.0028x + 0.4486

y = -0.0032x + 0.1783

y = -0.0026x + 0.0589

-0.15

-0.05

0.05

0.15

0.25

0.35

0.45

10 20 30 40 50 60Time (sec)

Au

to B

enef

it (

No

ne

- A

uto

)FullHazardToT & Fuel

Increasing Auto Benefit

14

Failure Analysis

• Most failures with hazard–Contrary to quantitative performance

• 40 sec had fewest failures–While no quantitative improvement in performance after 28 sec

• Failures at 55 sec 20 secFailure Count Time Automation 20 28 40 55 Grand Total

None 2 1 1 2 6ToT & Fuel 0 3 2 3 8

Hazard 6 2 0 5 13Full 2 1 1 1 5

Grand Total 10 7 4 11 32

A failed scenario had 1 of the following:1. Hit a brown hazard2. Arrived target outside of time window3. Not enough fuel to complete mission

0

1

2

3

4

5

0 1 2 3 4 5 6

Failure Count

Su

bje

ct F

req

uen

cy

• Only 1 subject was perfect• 14.3% failure rate, 32 of 224

15

Conclusions

• Automation does assist in time-critical decision making– Auto benefit decreases as available time

increases

– Auto benefit increases dependent on type and amount of information integrated by automation

• Moderate amounts of time may actually hinder performance over less time

• Subject data supported quantitative analysis

16

Future Work

• Develop a generalized model for decision support automation

• Identify information support needs of human

• Follow-on experiment– More specific to flight environment– Data points > 55 seconds to reach

characteristic time– Further test interactions between partial

and no automation