Upload
emil-summers
View
217
Download
0
Embed Size (px)
Citation preview
Reliance on Decision AidsReliance on Decision Aids
Results from 15 Studies Using a Soldier-Detection Task
April 12, 2006
CollaboratorsCollaborators• Linda PierceLinda Pierce• Hall (Skip) BeckHall (Skip) Beck• Scott PetersonScott Peterson• Regina PomrankyRegina Pomranky• Lori PurcellLori Purcell• Lloyd DaweLloyd Dawe• Catherine FieldCatherine Field• Angela PriceAngela Price
Cameron University
• Jennifer BatkaJennifer Batka• Emily BeadlesEmily Beadles• Paul ClarkPaul Clark• Tamara YoungTamara Young• Paul ClarkPaul Clark• Rachael RossRachael Ross• Adrian GohlAdrian Gohl• Emily DesorentoEmily Desorento
How can one best design How can one best design a system that will a system that will
encourage humans to encourage humans to appropriately use an appropriately use an
automated decision aid?automated decision aid?
Research QuestionResearch Question
Human Factors Literature:Human Factors Literature:Inappropriate Use of AutomationInappropriate Use of Automation
Parasuraman & Riley (1997)Parasuraman & Riley (1997)
Misuse: Misuse: Overreliance on automationOverreliance on automation
Disuse: Underutilization Disuse: Underutilization of automationof automation
Study 1Study 1
In Study 1, the reliability of the aid was In Study 1, the reliability of the aid was manipulated in a Visual Detection manipulated in a Visual Detection
Task and automation use was Task and automation use was examinedexamined
Study 1 Research DesignStudy 1 Research Design
• Four Conditions:Four Conditions:– AloneAlone– Computer with 90% Accuracy Computer with 90% Accuracy – Computer with 75% AccuracyComputer with 75% Accuracy– Computer with 60% AccuracyComputer with 60% Accuracy
• Dependent VariablesDependent Variables– PerformancePerformance
– p(incorrect)p(incorrect)– Inappropriate Automation UseInappropriate Automation Use
– Misuse = p(incorrect | aid incorrect)Misuse = p(incorrect | aid incorrect)– Disuse = p(incorrect | aid correct)Disuse = p(incorrect | aid correct)
p(US | CS)p(US | CS)
Inappropriate Automation UseInappropriate Automation UseRescorla-Wagner ModelRescorla-Wagner Model
p(US | no CS)p(US | no CS)
inhibitory association
excitatory association
no a
ssoc
iation
Inappropriate Automation UseInappropriate Automation Use
p(error | aid error)p(error | aid error)
p(p(errorerror | aid correct | aid correct))
Opposite ofAid
Reliance onAid
Igno
ring
of a
id
Procedure
Two Hundred Trials
View Slide
Make Decision
Indicate Confidence Level
View Aid’s Decision
Contrast Detector
Continue
PRESENT
Response Panel
Was the enemy in the precedingphotograph?
YES NO
Continue
Decision: YES
Continue
Highly Confident
Confident
Somewhat Confident
Slightly Confident
Not At All Confident
How confidentare you thatyou have madethe correctdecision?
AnalysisAnalysisSignal Detection TheorySignal Detection Theory
Soldier Soldier PRESENTPRESENT
Soldier Soldier ABSENTABSENT
Participant Participant responds responds PRESENTPRESENT
HitHit False AlarmFalse Alarm
Participant Participant responds responds ABSENTABSENT
MissMiss Correct Correct RejectionRejection
AnalysisAnalysisSignal Detection TheorySignal Detection Theory
Soldier Soldier PRESENTPRESENT
Soldier Soldier ABSENTABSENT
Participant Participant responds responds PRESENTPRESENT
HitHit False AlarmFalse Alarm
Participant Participant responds responds ABSENTABSENT
MissMiss Correct Correct RejectionRejection
AnalysisAnalysisSignal Detection TheorySignal Detection Theory
Soldier Soldier PRESENTPRESENT
Soldier Soldier ABSENTABSENT
Participant Participant responds responds PRESENTPRESENT
HitHit False AlarmFalse Alarm
Participant Participant responds responds ABSENTABSENT
MissMiss Correct Correct RejectionRejection
AnalysisAnalysisSignal Detection TheorySignal Detection Theory
Soldier Soldier PRESENTPRESENT
Soldier Soldier ABSENTABSENT
Participant Participant responds responds PRESENTPRESENT
HitHit False AlarmFalse Alarm
Participant Participant responds responds ABSENTABSENT
MissMiss Correct Correct RejectionRejection
If you use p(error), you miss potentially useful informationIf you use p(error), you miss potentially useful information
AnalysisAnalysisSignal Detection TheorySignal Detection Theory
Soldier Soldier PRESENTPRESENT
Soldier Soldier ABSENTABSENT
Participant Participant responds responds PRESENTPRESENT
HitHit False AlarmFalse Alarm
Participant Participant responds responds ABSENTABSENT
MissMissCorrect Correct
RejectionRejection
TotalTotal Signal TotalSignal Total Noise TotalNoise Total
AnalysisAnalysisSignal Detection TheorySignal Detection Theory
Soldier Soldier PRESENTPRESENT
Soldier Soldier ABSENTABSENT
Participant Participant responds responds PRESENTPRESENT
HitHit False AlarmFalse Alarm
TotalTotal Signal TotalSignal Total Noise TotalNoise Total
Hit Rate = p(S says present | soldier is present)Hit Rate = p(S says present | soldier is present)False Alarm Rate = p(S says present | soldier is absent)False Alarm Rate = p(S says present | soldier is absent)
AnalysisAnalysisSignal Detection TheorySignal Detection Theory
• Subject’s SensitivitySubject’s Sensitivity– How well the participant discriminated between the How well the participant discriminated between the
soldier present and absent trialssoldier present and absent trials– A subject with PERFECT sensitivity would haveA subject with PERFECT sensitivity would have
• Hit Rate = 1Hit Rate = 1• False Alarm Rate = 0False Alarm Rate = 0
– A subject with NO sensitivityA subject with NO sensitivity• Hit Rate = False Alarm RateHit Rate = False Alarm Rate
– Usually subjects fall between these two extremesUsually subjects fall between these two extremes– d’ = z(Hit Rate) – z(False Alarm Rate)d’ = z(Hit Rate) – z(False Alarm Rate)
AnalysisAnalysisSignal Detection TheorySignal Detection Theory
Soldier AbsentSoldier Absent Soldier PresentSoldier Present
What is d’?What is d’?
AnalysisAnalysisSignal Detection TheorySignal Detection Theory
Soldier AbsentSoldier Absent Soldier PresentSoldier Present
d’d’
What is d’?What is d’?
AnalysisAnalysisSignal Detection TheorySignal Detection Theory
Soldier AbsentSoldier Absent Soldier PresentSoldier Present
What is bias?What is bias?C = -.5[z(Hit Rate) + z(False Alarm Rate)]C = -.5[z(Hit Rate) + z(False Alarm Rate)]
C=0C=0c = participants bias to say absent or presentc = participants bias to say absent or present
AnalysisAnalysisSignal Detection TheorySignal Detection Theory
ROC Curve (Receiver Operating Characteristic)ROC Curve (Receiver Operating Characteristic)
False Alarm RateFalse Alarm Rate
Hit
Rat
eH
it R
ate
d’=0d’=0d’=.5d’=.5
0
0.5
1
1.5
2
2.5
3
3.5
-2.1 -1.6 -1.1 -0.6 -0.1 0.4 0.9 1.4
Hits
Fal
se A
larm
s
ROC of Standardized Cumulative Probabilities ROC of Standardized Cumulative Probabilities for Subject #37 for Trials When Machine is Correctfor Subject #37 for Trials When Machine is Correct
A(z) = the complement of the area under the ROC curveA(z) = the complement of the area under the ROC curve
Study 1 ResultsOverall Performance
0
0.1
0.2
0.3
0.4
0.5
Alone 60% 75% 90%
ProbabilityProbabilityIncorrectIncorrect
A(z)A(z)
ConditionCondition
Study 1 ResultsExtent of Automation Use
DISUSEDISUSEProbability ofProbability of
Incorrectly Incorrectly IgnoringIgnoring
the Systemthe System
p(incorrect | aid correct)p(incorrect | aid correct)
MISUSEMISUSEProbability ofProbability of
Incorrectly Relying Incorrectly Relying onon
the Systemthe System
p(incorrect | aid incorrect)p(incorrect | aid incorrect)
.128.128 .265.265
Study 1 ResultsEffect of Reliability on Automation Use
0.21
0.13
0.25
0.12
0.34
0.13
0
0.1
0.2
0.3
0.4
0.5
60% 75% 90%
Aid Incorrect Aid Correct
Reliability of Automated AidReliability of Automated Aid
ProbabilityProbabilityIncorrectIncorrect
A(z)A(z)
Conclusions from Study 1Conclusions from Study 1
• Machine accuracy (or automation reliability) did Machine accuracy (or automation reliability) did not affect reliance on automated aid or overall not affect reliance on automated aid or overall performanceperformance
• Participants in the aided conditions relied on the Participants in the aided conditions relied on the aid more than they ignored the aidaid more than they ignored the aid
Study 1 ImplicationsStudy 1 Implications
• BottomBottom Line• Improving the reliability of the automated decision Improving the reliability of the automated decision
aid does NOT improve the performance of the aid does NOT improve the performance of the human-automation teamhuman-automation team
• People relied on the unreliable decision aid almost People relied on the unreliable decision aid almost as much as they relied on the reliable decision as much as they relied on the reliable decision aid...aid...
• QuestionQuestion• Why do people inappropriately use automated Why do people inappropriately use automated
aids?aids?
Why do people inappropriately useautomated aids?
Group DynamicsGroup DynamicsLiteratureLiterature
Human FactorsHuman FactorsLiteratureLiterature
Cognitive Cognitive ProcessesProcesses
Cognitive MiserCognitive MiserHypothesisHypothesis
MotivationalMotivationalProcessesProcesses
Diffusion ofDiffusion ofResponsibilityResponsibility
Social ProcessesSocial Processes Authority HypothesisAuthority Hypothesis
Cognitive Processes
• People use the information provided by the automated aid in a heuristic manner
• The automated aid’s decision may be given undue weight in the human’s decision making process
Model of Automation Reliance:Cognitive Processes
Cognitive ProcessesCognitive Processes
Provision of Aid’sProvision of Aid’sDecision Prior toDecision Prior toDecision MakingDecision Making
AutomationAutomationUseUse
To eliminate this process, provide the aid’s decisionTo eliminate this process, provide the aid’s decisiononly after the participant has indicated his decision.only after the participant has indicated his decision.
Motivational Processes:Diffusion of Responsibility
Shepperd’s Expectancy-Value Theory
• Expectancy• Instrumentality• Outcome Value
Motivation is a function ofMotivation is a function of
Motivational Processes:Shepperd’s Expectancy-Value Theory
• Expectancy – The extent to which members feel their
efforts are necessary for the group to succeed
• Instrumentality– The extent to which members feel that the
group’s successful performance will lead to a positive outcome
• Outcome Value – Importance of Outcome - Costs of Outcome
Model of Automation Reliance:Motivational Processes
Motivational ProcessesMotivational Processes
AutomationAutomationUseUseEffortEffort
ExpectancyExpectancy
DispensabilityDispensability
Perceived UtilityPerceived Utilityof Automated Aidof Automated Aid
Task DifficultyTask Difficulty
InstrumentalityInstrumentality
Outcome ValueOutcome ValueCostsCosts
ImportanceImportance
WorkloadWorkload
FatigueFatigue
Intrinsic InterestIntrinsic Interest in the Taskin the Task
PenaltiesPenalties
RewardsRewards To eliminate this process, make the effort expendedTo eliminate this process, make the effort expendedfor self-reliance equal to that for automation reliance.for self-reliance equal to that for automation reliance.
Motivational Processes
AutomationRelianceEffort
Expectancy
Dispensability
Perceived Utilityof Automated Aid
Task Difficulty
Instrumentality
Outcome ValueCosts
Importance
Workload
Fatigue
Intrinsic Interest in the Task
Penalties
Rewards
Social Processes:Authority Hypothesis
TRUSTTRUST
Model of Automation Use:Social Processes
Automation Automation UseUse
Social ProcessesSocial Processes
RelativeRelative TrustTrust
PerceivedPerceivedUtility ofUtility of
AutomatedAutomatedAidAid
Cognitive ProcessesCognitive Processes
Perceived ReliabilityPerceived Reliabilityof Automatedof Automated
AidAid
Perceived ReliabilityPerceived Reliabilityof Manual of Manual OperationOperation
ActualActualReliabilityReliability
of of AutomatedAutomated
AidAid
ActualActualReliabilityReliability
of of Manual Manual
OperationOperation
Bias TowardBias TowardAutomationAutomation
Self-Serving BiasSelf-Serving Bias
Study 2Study 2
Study 2 examined social processes (TRUST)Study 2 examined social processes (TRUST)while eliminating the cognitive processes while eliminating the cognitive processes
and holding constant the motivational and holding constant the motivational processesprocesses
Procedure
Two Hundred Trials
View Slide
Make Decision
Indicate Confidence Level
View Aid’s Decision
Response Panel
Was the enemy in the precedingphotograph?
YES NO
Continue
Decision: YES
Continue
Highly Confident
Confident
Somewhat Confident
Slightly Confident
Not At All Confident
How confidentare you thatyou have madethe correctdecision?
View Feedback (some participants)
Contrast Detector
Continue
PRESENT
Study 2 DesignStudy 2 Design
• Independent VariablesIndependent Variables– Partner’s Performance LevelPartner’s Performance Level– Provision of Cumulative FeedbackProvision of Cumulative Feedback– Reward Based on Future Trials…Reward Based on Future Trials…
InstructionsInstructions
……You will perform an additional ten trials. The amount You will perform an additional ten trials. The amount of extra credit that you will earn will entirely depend of extra credit that you will earn will entirely depend on the outcome of these next ten trials…on the outcome of these next ten trials…
The amount of extra credit...will either be based on The amount of extra credit...will either be based on your performance or on the performance of the your performance or on the performance of the contrast detector. It is your choice.contrast detector. It is your choice.
Study 2 DesignStudy 2 Design
• Independent VariablesIndependent Variables– Partner’s Performance LevelPartner’s Performance Level– Provision of Cumulative FeedbackProvision of Cumulative Feedback– Reward Based on Future or Past TrialsReward Based on Future or Past Trials
InstructionsInstructions
……Ten trials will be randomly selected from Ten trials will be randomly selected from the preceding 200. The amount of extra creditthe preceding 200. The amount of extra creditthat you will earn will entirely depend upon the that you will earn will entirely depend upon the outcome of those trials.…outcome of those trials.…
The amount of extra credit...will either be based on The amount of extra credit...will either be based on your performance or on the performance of the your performance or on the performance of the contrast detector. It is your choice.contrast detector. It is your choice.
Study 2 DesignStudy 2 Design
• Independent VariablesIndependent Variables– Partner’s Performance LevelPartner’s Performance Level– Provision of Cumulative FeedbackProvision of Cumulative Feedback– Reward Based on Future or Past TrialsReward Based on Future or Past Trials
• Dependent VariableDependent Variable– Choice of Self or Aid’s Performance to Determine Choice of Self or Aid’s Performance to Determine
RewardReward
Study 2 HypothesesStudy 2 Hypotheses
• Those working with an aid superior to the Those working with an aid superior to the participant would trust the automated systemparticipant would trust the automated system
• Those working with an aid inferior to the participant Those working with an aid inferior to the participant would not trust the automated systemwould not trust the automated system
• If self-serving biases exist, above pattern of results If self-serving biases exist, above pattern of results would appear only among participants given would appear only among participants given feedbackfeedback
Study 2 ResultsStudy 2 Results
ConditionCondition SelfSelf AidAid
Future Trials-No Feedback-InferiorFuture Trials-No Feedback-InferiorFuture Trials-No Feedback-EqualFuture Trials-No Feedback-EqualFuture Trials-No Feedback-Future Trials-No Feedback-SuperiorSuperior
998888
001111
Study 2 ResultsStudy 2 Results
ConditionCondition SelfSelf AidAid
Future Trials-No Feedback-InferiorFuture Trials-No Feedback-InferiorFuture Trials-No Feedback-EqualFuture Trials-No Feedback-EqualFuture Trials-No Feedback-Future Trials-No Feedback-SuperiorSuperiorFuture Trials-Feedback-InferiorFuture Trials-Feedback-InferiorFuture Trials-Feedback-EqualFuture Trials-Feedback-EqualFuture Trials-Feedback-SuperiorFuture Trials-Feedback-Superior
998888999999
001111000000
Study 2 ResultsStudy 2 Results
ConditionCondition SelfSelf AidAid
Future Trials-No Feedback-InferiorFuture Trials-No Feedback-InferiorFuture Trials-No Feedback-EqualFuture Trials-No Feedback-EqualFuture Trials-No Feedback-Future Trials-No Feedback-SuperiorSuperiorFuture Trials-Feedback-InferiorFuture Trials-Feedback-InferiorFuture Trials-Feedback-EqualFuture Trials-Feedback-EqualFuture Trials-Feedback-SuperiorFuture Trials-Feedback-SuperiorPast Trials-No Feedback-InferiorPast Trials-No Feedback-InferiorPast Trials-No Feedback-EqualPast Trials-No Feedback-EqualPast Trials-No Feedback-SuperiorPast Trials-No Feedback-Superior
998888999999998899
001111000000001100
Study 2 ResultsStudy 2 Results
ConditionCondition SelfSelf AidAid
Future Trials-No Feedback-InferiorFuture Trials-No Feedback-InferiorFuture Trials-No Feedback-EqualFuture Trials-No Feedback-EqualFuture Trials-No Feedback-Future Trials-No Feedback-SuperiorSuperiorFuture Trials-Feedback-InferiorFuture Trials-Feedback-InferiorFuture Trials-Feedback-EqualFuture Trials-Feedback-EqualFuture Trials-Feedback-SuperiorFuture Trials-Feedback-SuperiorPast Trials-No Feedback-InferiorPast Trials-No Feedback-InferiorPast Trials-No Feedback-EqualPast Trials-No Feedback-EqualPast Trials-No Feedback-SuperiorPast Trials-No Feedback-SuperiorPast Trials-Feedback-InferiorPast Trials-Feedback-InferiorPast Trials-Feedback-EqualPast Trials-Feedback-EqualPast Trials-Feedback-SuperiorPast Trials-Feedback-Superior
998888999999998899998866
001111000000001100001133
Conclusions from Study 2Conclusions from Study 2
• When controlling motivational and cognitive When controlling motivational and cognitive processes, find disuse of automated aidprocesses, find disuse of automated aid
• Even when given feedback that the computer was Even when given feedback that the computer was superior, most participants chose to rely on their superior, most participants chose to rely on their performanceperformance
Study 2 ImplicationsStudy 2 Implications
• Bottom LineBottom Line• In this paradigm, people did In this paradigm, people did
NOT appropriately rely on a NOT appropriately rely on a superior automated decision superior automated decision aidaid
• QuestionQuestion• Are people biased against Are people biased against
automated decision aids?automated decision aids?
Study 3 DesignStudy 3 Design
• Independent VariablesIndependent Variables– Partner’s Performance LevelPartner’s Performance Level– Provision of Feedback Provision of Feedback – Type of PartnerType of Partner
• Dependent VariableDependent Variable– Choice of Self or Partner’s Performance during 10 Choice of Self or Partner’s Performance during 10
Randomly Chosen PAST Trials to Determine RewardRandomly Chosen PAST Trials to Determine Reward
Study 3 ProcedureStudy 3 Procedure
• Two Hundred TrialsTwo Hundred Trials– View SlideView Slide– Make DecisionMake Decision– Indicate Confidence LevelIndicate Confidence Level– View Other’s DecisionView Other’s Decision
• View Feedback/InstructionsView Feedback/Instructions• Decide Reward BasisDecide Reward Basis
InstructionsInstructions
……Ten trials will be randomly selected from Ten trials will be randomly selected from the preceding 200. The amount of money (in the preceding 200. The amount of money (in coupons) that you will obtain will entirely coupons) that you will obtain will entirely depend upon the outcome of those trials.…depend upon the outcome of those trials.…
The amount of money...will either be based The amount of money...will either be based on your performance or on the performanceon your performance or on the performanceof the prior student. It is your choice.of the prior student. It is your choice.
Study 3 ResultsStudy 3 Results
00
2020
4040
6060
8080
100100
FdbkFdbk No FdbkNo Fdbk FdbkFdbk No FdbkNo FdbkSuperiorSuperior EqualEqual
Human AidHuman AidAutomated AidAutomated Aid
PercentPercentChose Chose to Rely to Rely on Selfon Self
chancechance
Study 3 ResultsStudy 3 Results
00
2020
4040
6060
8080
100100
FdbkFdbk No FdbkNo Fdbk FdbkFdbk No FdbkNo FdbkSuperiorSuperior EqualEqual
PercentPercentChose Chose to Rely to Rely on Selfon Self
chancechance
Human AidHuman AidAutomated AidAutomated Aid
Conclusions from Study 3Conclusions from Study 3
• As in Study 2, a strong bias to rely on self was found As in Study 2, a strong bias to rely on self was found when cognitive and motivational processes were when cognitive and motivational processes were controlledcontrolled
• Evidence of a bias against automationEvidence of a bias against automation– In the feedback-superior condition: In the feedback-superior condition:
53% participants relied on a human aid53% participants relied on a human aid<6% participants relied on the automated aid<6% participants relied on the automated aid
Why did you choose to rely on aid?Why did you choose to rely on aid?Cognitive Processes: Logical ExplanationsCognitive Processes: Logical Explanations
00
22
44
66
88
1010
FdbkFdbk No FdbkNo Fdbk FdbkFdbk No FdbkNo Fdbk SuperiorSuperior Equal Equal
NumberNumberofof
LogicalLogicalStatementsStatements
ConditionConditionHuman AidHuman AidAutomated AidAutomated Aid
Why did you choose to rely on yourself?Why did you choose to rely on yourself?Social Processes: Assessing ReliabilitySocial Processes: Assessing Reliability
0
1
2
3
4
5
6
Fdbk No Fdbk Fdbk No Fdbk
SuperiorSuperior Equal Equal
ConditionCondition
NumberNumberofof
StatementsStatementsAssessing Assessing ReliabilityReliability
Human AidHuman AidAutomated AidAutomated Aid
Why did you choose to rely on yourself?Why did you choose to rely on yourself?Social Processes: Relative TrustSocial Processes: Relative Trust
012345678
Fdbk No Fdbk Fdbk No Fdbk
SuperiorSuperior Equal Equal
NumberNumberofof
CommentsCommentsConcerningConcerning
RelativeRelativeTrustTrust
ConditionConditionHuman AidHuman AidAutomated AidAutomated Aid
Why did you choose to rely on yourself?Why did you choose to rely on yourself?Cognitive Processes: Aid Made a MistakeCognitive Processes: Aid Made a Mistake
0
2
4
6
8
10
12
Fdbk No Fdbk Fdbk No Fdbk
SuperiorSuperior Equal Equal
NumberNumberofof
CommentsCommentsConcerningConcerning
Errors Errors Made byMade by
AidAid
ConditionConditionHuman AidHuman AidAutomated AidAutomated Aid
Conclusions from Study 3Conclusions from Study 3
• Consistent with the model, people reported Consistent with the model, people reported assessing the reliability of the aid, their own ability, assessing the reliability of the aid, their own ability, and the difference between the twoand the difference between the two
• People remembered mistakes made by automated People remembered mistakes made by automated aids more than they remembered mistakes made by aids more than they remembered mistakes made by human aids and used this information to affect their human aids and used this information to affect their decision to rely on the aiddecision to rely on the aid
Study 3 ImplicationsStudy 3 Implications
• Bottom LineBottom Line• People were more likely to appropriately rely on a People were more likely to appropriately rely on a
human aid than on an automated decision aidhuman aid than on an automated decision aid• They were more likely to justify their disuse of the They were more likely to justify their disuse of the
automated aid than the human aid by describing automated aid than the human aid by describing instances in which they knew the aid made an errorinstances in which they knew the aid made an error
• QuestionQuestion• How well do participants expect the automated How well do participants expect the automated
decision aid to perform?decision aid to perform?
Study 4 ProcedureStudy 4 Procedure
• Told They Will Perform Two Hundred TrialsTold They Will Perform Two Hundred Trials– View SlideView Slide– View Other’s DecisionView Other’s Decision– Make DecisionMake Decision– Indicate Confidence LevelIndicate Confidence Level
• Perform Four Practice SlidesPerform Four Practice Slides
Study 4 DesignStudy 4 Design
• Independent VariableIndependent Variable– Type of AidType of Aid
– Prior ParticipantPrior Participant– Contrast DetectorContrast Detector
• Dependent VariableDependent Variable– Estimated Number of Errors in Upcoming 200 TrialsEstimated Number of Errors in Upcoming 200 Trials
Study 4 ResultsStudy 4 Results
0
10
20
30
40
Automated Aid Human Aid
EstimatedErrors
Type of PartnerType of PartnerAidSelf
How many errors do you think the contrast How many errors do you think the contrast detector/prior participant will make? (blank line)detector/prior participant will make? (blank line)
Study 4 ImplicationsStudy 4 Implications
• Bottom LineBottom Line• Participants expect automated aids to perform Participants expect automated aids to perform
better than human aidsbetter than human aids
• QuestionQuestion• What happens during the interaction with the What happens during the interaction with the
automated system that turns the bias toward automated system that turns the bias toward automation into a bias against automation?automation into a bias against automation?
Perhaps...
Expectations
Inconsistent
Very well rememberedSmith & Graesser (1981)
Inaccurate Relative Perception
Study 5 DesignStudy 5 Design
• Independent VariablesIndependent Variables– Framing of Past PerformanceFraming of Past Performance
– Positively (“Aid makes 1/2 as many errors as most students”)Positively (“Aid makes 1/2 as many errors as most students”)– Negatively (“Aid makes 10 errors”)Negatively (“Aid makes 10 errors”)– Unspecified (“Aid is not perfect”)Unspecified (“Aid is not perfect”)
– Type of PartnerType of Partner
• Dependent VariableDependent Variable– Choice of Self or Partner’s Performance during 10 Choice of Self or Partner’s Performance during 10
Randomly Chosen PAST Trials to Determine RewardRandomly Chosen PAST Trials to Determine Reward
Study 5 ProcedureStudy 5 Procedure
• Two Hundred TrialsTwo Hundred Trials– View SlideView Slide
– Make DecisionMake Decision
– Indicate Confidence LevelIndicate Confidence Level
– View Other’s DecisionView Other’s Decision
• View FeedbackView Feedback
• Decide Reward BasisDecide Reward Basis
Study 5 ResultsStudy 5 Results
0
20
40
60
80
100
Negatively Positively Unspecified
Human Aid Automated Aid
Self-
Rel
ianc
eSe
lf-R
elia
nce
Conclusions from Study 5Conclusions from Study 5
• Consistent with past research, disuse was commonConsistent with past research, disuse was common
• Consistent with “Perfect Automation Schema” Consistent with “Perfect Automation Schema” hypothesis, those who expected a less perfect hypothesis, those who expected a less perfect performance from their automated decision aid performance from their automated decision aid were more likely to rely on the aidwere more likely to rely on the aid
Study 5 ImplicationsStudy 5 Implications
• Bottom LineBottom Line• People are more likely to appropriately rely on an People are more likely to appropriately rely on an
automated decision aid if the aid’s errors are salient…automated decision aid if the aid’s errors are salient…• Reducing people’s initial trust in an automated Reducing people’s initial trust in an automated
decision aid may lead to more appropriate reliancedecision aid may lead to more appropriate reliance
• QuestionQuestion• What other types of intervention will lead to more What other types of intervention will lead to more
appropriate automation reliance?appropriate automation reliance?
Study 6 ProcedureStudy 6 Procedure
• Scenario TrainingScenario Training
• Two Hundred TrialsTwo Hundred Trials– View SlideView Slide
– Make DecisionMake Decision
– Indicate Confidence LevelIndicate Confidence Level
– View Other’s DecisionView Other’s Decision
• View Feedback (5 seconds)View Feedback (5 seconds)
• Decide Reward BasisDecide Reward Basis
Scenario TrainingScenario Training
• A stockbroker was giving investment advice. Research showed an average gain of 7% per year, when the broker made suggestions based on personal study. In contrast, recommendations made from a computer program yielded an annual average gain of 4%.
• Is the better investment strategy to rely on recommendations based on the stockbroker’s own study or on the computer program?
Study 6 ResultsStudy 6 Results
0102030405060708090
100
Fdbk No Fdbk Fdbk No Fdbk
Scenario No Scenario
Sel
f-R
elia
nce
Superior InferiorSuperior Inferior
Study 6 ImplicationsStudy 6 Implications
• Bottom LineBottom Line• People are more likely to appropriately rely on a People are more likely to appropriately rely on a
superior automated decision aid if they are given superior automated decision aid if they are given some training about probability.some training about probability.
• QuestionQuestion• What other types of intervention will lead to more What other types of intervention will lead to more
appropriate automation reliance?appropriate automation reliance?• Will feedback reduce disuse of automation? Will feedback reduce disuse of automation?
Study 7 DesignStudy 7 Design
• Independent VariablesIndependent Variables– Cumulative Results FeedbackCumulative Results Feedback– Individual Trial FeedbackIndividual Trial Feedback– Prior Results FeedbackPrior Results Feedback
• Dependent VariableDependent Variable– Choice of Self or Partner’s Performance during 10 Choice of Self or Partner’s Performance during 10
Randomly Chosen PAST Trials to Determine RewardRandomly Chosen PAST Trials to Determine Reward
Study 7 ProcedureStudy 7 Procedure
• Two Hundred TrialsTwo Hundred Trials– View SlideView Slide
– Make DecisionMake Decision
– Indicate Confidence LevelIndicate Confidence Level
– View Other’s DecisionView Other’s Decision
– View Feedback (some participants)View Feedback (some participants)
• View Feedback View Feedback (some participants)(some participants)
• Prior Results Feedback Prior Results Feedback (some participants)(some participants)
• Decide Reward BasisDecide Reward Basis
Study 7 ResultsStudy 7 Results
0
20
40
60
80
100
0 1 2 3
Self
- Rel
ianc
eSe
lf - R
elia
nce
Number of Sources of FeedbackNumber of Sources of Feedback
Study 7 ImplicationsStudy 7 Implications
• Bottom LineBottom Line• Providing participants with various types of Providing participants with various types of
information concerning their and their information concerning their and their automated aid’s performance reduces the automated aid’s performance reduces the bias toward disusebias toward disuse
• QuestionQuestion What other techniques will reduce disuse of What other techniques will reduce disuse of
automated aids?automated aids?
Study 8 DesignStudy 8 Design
• Independent VariablesIndependent Variables– Provision of Aid’s DecisionProvision of Aid’s Decision– Aid’s Performance LevelAid’s Performance Level– Type of FeedbackType of Feedback
• ContinuousContinuous
0
5
10
15
20
25
30
35
# of Errors Made
Contrast Detector
YOU
Continuous FeedbackContinuous Feedback
Remains on screen Remains on screen during all 200 trialsduring all 200 trials
Increases accuracyIncreases accuracyof perceived utilityof perceived utility
Updated every 5Updated every 5trialstrials
EliminatesEliminatesdetection of detection of
obvious errorsobvious errors
Study 8 DesignStudy 8 Design
• Independent VariablesIndependent Variables– Provision of Aid’s DecisionProvision of Aid’s Decision– Aid’s Performance LevelAid’s Performance Level– Type of FeedbackType of Feedback
• ContinuousContinuous• CumulativeCumulative• NoneNone
• Dependent VariableDependent Variable– Choice of Self or Partner’s Performance during 10 Randomly Choice of Self or Partner’s Performance during 10 Randomly
Chosen PAST Trials to Determine RewardChosen PAST Trials to Determine Reward
Study 8 Performance DataStudy 8 Performance Data
0
20
40
60
80
SuperiorSuperior
No Feedback No Decision Continuous Feedback Decision
No Feedback Decision Cumulative Feedback Decision
Cumulative Feedback No Decision Continuous Feedback No Decision
Dis
use
Providing Continuous Feedback andProviding Continuous Feedback andPreventing Participants from Viewing Obvious Preventing Participants from Viewing Obvious Errors Made by the Automated Aid Reduced Errors Made by the Automated Aid Reduced
Disuse without Increasing MisuseDisuse without Increasing Misuse
0
20
40
60
80
100
SuperiorSuperior InferiorInferior
% o
f th
ose
wh
o
cho
seto
rel
y o
n s
elf
Study 8 ResultsStudy 8 ResultsJustificationsJustifications
• 85% of all justifications fell into one of four categories85% of all justifications fell into one of four categories• Trust in computersTrust in computers
– I don’t trust computers that much. I know a lot about their I don’t trust computers that much. I know a lot about their tendency for errorstendency for errors
• Detection of obvious errorsDetection of obvious errors– There were a few times that I’m pretty sure I saw the soldier, There were a few times that I’m pretty sure I saw the soldier,
but the program said he was absentbut the program said he was absent• Confidence in selfConfidence in self
– I was not confident in what I sawI was not confident in what I saw• Relative performanceRelative performance
– The computer made more mistakes compared to mineThe computer made more mistakes compared to mine
Study 8 ResultsStudy 8 Results
• Questions asked to Decision PresentQuestions asked to Decision Present– How much did you trust the decisions made by the How much did you trust the decisions made by the
contrast detector (very little – a great amount)contrast detector (very little – a great amount)• Overall, below the midpointOverall, below the midpoint• Superior aids were rated more trustworthy than inferior Superior aids were rated more trustworthy than inferior
aidsaids– To what extent did you lose trust in the contrast To what extent did you lose trust in the contrast
detector when you noticed it made an error? (very detector when you noticed it made an error? (very little – a great amount)little – a great amount)
• Overall, above the midpointOverall, above the midpoint
Study 8 ResultsStudy 8 Results
• Questions asked to Decision PresentQuestions asked to Decision Present– Compared with those who chose to rely on self, Compared with those who chose to rely on self,
those who chose to rely on the automated decision those who chose to rely on the automated decision aidaid
– Trusted the aid moreTrusted the aid more– Noticed the aid made fewer errorsNoticed the aid made fewer errors– Indicated they lost LESS trust in the aid when they Indicated they lost LESS trust in the aid when they
noticed an error the aid madenoticed an error the aid made
Study 8 ImplicationsStudy 8 Implications • Bottom LineBottom Line• Providing participants with continuous feedback and Providing participants with continuous feedback and
eliminating the possible detection of obvious errors drastically eliminating the possible detection of obvious errors drastically reduces disuse without causing misusereduces disuse without causing misuse
• More theoretical than appliedMore theoretical than applied• Trust may play a role (Lee & Moray’s (1992) performance, Trust may play a role (Lee & Moray’s (1992) performance,
process, and purpose)process, and purpose)
• QuestionQuestion• Will providing a reason as to why the aid might err lead Will providing a reason as to why the aid might err lead
to more appropriate reliance? to more appropriate reliance?
Study 9 DesignStudy 9 Design
• Independent VariableIndependent Variable– ReasonReason– RationaleRationale
• Dependent VariableDependent Variable– Choice of Self or Partner’s Performance during 10 Choice of Self or Partner’s Performance during 10
Randomly Chosen PAST Trials to Determine RewardRandomly Chosen PAST Trials to Determine Reward
I need to inform you that this machine is I need to inform you that this machine is NOTNOT perfect. The contrast detector will indicate the perfect. The contrast detector will indicate the soldier is present if it detects a form that humans soldier is present if it detects a form that humans often take. often take. Since non-humans (e.g. shading Since non-humans (e.g. shading from a tree) can sometimes take human-like from a tree) can sometimes take human-like forms, mistakes can be made.forms, mistakes can be made. Therefore, if your Therefore, if your decision and the decision reached by the program decision and the decision reached by the program are not the same, do NOT assume that your decision are not the same, do NOT assume that your decision is incorrect.is incorrect.
ReasonReason
Study 9 ProcedureStudy 9 Procedure
• Two Hundred TrialsTwo Hundred Trials– View SlideView Slide
– Make DecisionMake Decision
– Indicate Confidence LevelIndicate Confidence Level
– View Other’s DecisionView Other’s Decision
• View Feedback that Aid Is SuperiorView Feedback that Aid Is Superior
• Decide Reward BasisDecide Reward Basis
Study 9 Results Study 9 Results
0
5
10
15
20
25
30
35
40
45
No Reason Reason No Reason Reason
Dis
use
No Rationale RationaleNo Rationale Rationale
Study 9 ImplicationsStudy 9 Implications
• Bottom LineBottom Line• In this artificial paradigm (in which the automation bias is In this artificial paradigm (in which the automation bias is
eliminated and the expended effort is controlled), neither eliminated and the expended effort is controlled), neither providing participants with a reason (for false alarms) nor a providing participants with a reason (for false alarms) nor a rationale for why an aid might err reduced disuserationale for why an aid might err reduced disuse
• QuestionQuestion• Would providing a reason as to why the aid might err lead to Would providing a reason as to why the aid might err lead to
more appropriate reliance in a more realistic paradigm (i.e., more appropriate reliance in a more realistic paradigm (i.e., one in which the automation bias is not eliminated and effort one in which the automation bias is not eliminated and effort is not controlled)? is not controlled)?
Study 10 DesignStudy 10 Design
• Independent Variables Independent Variables – Aid’s Relative PerformanceAid’s Relative Performance
– Provision of a Reason for Aid’s ErrorsProvision of a Reason for Aid’s Errors • Dependent VariablesDependent Variables
– Reliance on Aid:Reliance on Aid:– Transformed p(incorrect | incorrect)Transformed p(incorrect | incorrect)– Transformed p(incorrect | correct)Transformed p(incorrect | correct)
Study 10 ProcedureStudy 10 Procedure
• Two Hundred TrialsTwo Hundred Trials– View SlideView Slide
– Make DecisionMake Decision
– Indicate Confidence LevelIndicate Confidence Level
– View Other’s DecisionView Other’s Decision
• View FeedbackView Feedback– Inferior AidInferior Aid
– Superior AidSuperior Aid
• One Hundred TrialsOne Hundred Trials– View SlideView Slide– View Other’s DecisionView Other’s Decision– Make DecisionMake Decision– Indicate Confidence LevelIndicate Confidence Level
• Complete QuestionnaireComplete Questionnaire
Study 10 ResultsStudy 10 Results
00.20.40.60.8
11.21.41.6
No Reason Reason No Reason Reason
Aid Incorrect
Aid Correct
tran
sfor
med
tran
sfor
med
p(e
rror
)p(
erro
r)
Target Absent Target PresentTarget Absent Target Present
Study 10 ResultsStudy 10 ResultsTrust in Aid is Greater for Trust in Aid is Greater for
those Working with a Superior Aidthose Working with a Superior Aid
1
2
3
4
5
6
7
Superior Inferior
Self-
Rep
orte
d Tr
ust
Self-
Rep
orte
d Tr
ust
Study 10 ResultsStudy 10 ResultsTrust in Aid is Greater for Trust in Aid is Greater for
Those Provided with a ReasonThose Provided with a Reason
1
2
3
4
5
6
7
No Reason Reason
Self-
Rep
orte
d Tr
ust
Self-
Rep
orte
d Tr
ust
Three Conditions to Show Trust Mediates the Three Conditions to Show Trust Mediates the Relationship between Reason and RelianceRelationship between Reason and Reliance
Baron and Kenny (1986)Baron and Kenny (1986)
– Effect of reason on automation reliance is significantEffect of reason on automation reliance is significant– b1 = .55; t(1) = 2.39, p < .03 b1 = .55; t(1) = 2.39, p < .03
– Effect of reason on trust is significantEffect of reason on trust is significant– b1 = 1.30; t(1) = 2.12, p < .05b1 = 1.30; t(1) = 2.12, p < .05
– When controlling for trust, the effect of reason on When controlling for trust, the effect of reason on automation reliance is reducedautomation reliance is reduced– b1 = .52; t(1) = 1.81, p > .05b1 = .52; t(1) = 1.81, p > .05
Study 10 ImplicationsStudy 10 Implications • Bottom LineBottom Line• People were more likely to rely on a superior automated decision People were more likely to rely on a superior automated decision
aid if they were provided with a reason why it might err aid if they were provided with a reason why it might err • The relationship between reason and reliance was mediated by The relationship between reason and reliance was mediated by
trust trust – Those provided with a reason why the aid might err trusted the aid moreThose provided with a reason why the aid might err trusted the aid more– This increased trust led them to rely on the aid more oftenThis increased trust led them to rely on the aid more often
• QuestionQuestion• Will training in the conditions in which the aid is reliable and Will training in the conditions in which the aid is reliable and
unreliable lead to appropriate reliance?unreliable lead to appropriate reliance?
Individual CombatIdentification System
ICIDS
Battlefield Combat Identification System
BCIS
If the target returns a signal, soldiers will
receive the message: “FRIENDLYFRIENDLY”
Combat Identification SystemsCombat Identification Systems
If the target does not return a signal, soldiers
will receive the message:
“UNKNOWNUNKNOWN”
Absent 50 times
100%
50%
50%
Contrast Detector
Continue
ABSENT
Contrast Detector
Continue
PRESENT
Aid Says
Study 11 TrainingStudy 11 Training
Reality
Present 50 times
Absent
Present
Reality
Contrast Detector
Continue
PRESENT
Contrast Detector
Continue
ABSENT
Aid Said
75 times
25 times
100%
66.6%
33.3%
25 times
50 times
25 times
Study 11 TrainingStudy 11 Training
05
101520253035404550
Aid Correctly says ABSENT
Aid Incorrectlysays PRESENT
Aid Correctlysays PRESENT
Alone No Training Training
% o
f R
eloo
ksStudy 11 ResultsStudy 11 Results
0
0.1
0.2
0.3
0.4
0.5
TargetPresent
(MISUSE)
TargetAbsent
(DISUSE)
TargetAbsent
(MISUSE)
TargetPresent
(DISUSE)
Alone No Training Training
Aid CorrectlySays ABSENT
Aid SaysPRESENT
Tra
ns
form
ed p
(err
or)
Study 11 ResultsStudy 11 Results
Study 11 ImplicationsStudy 11 Implications
• Bottom LineBottom Line• Those provided with the automated decision aid Those provided with the automated decision aid
performed WORSE than those without itperformed WORSE than those without it• Misuse exceeded disuse--even when the reliability of Misuse exceeded disuse--even when the reliability of
the automated system was very low!the automated system was very low!• Training of the conditions in which the aid would be Training of the conditions in which the aid would be
reliable did little to improve appropriate reliance on reliable did little to improve appropriate reliance on the simple automated decision aidthe simple automated decision aid
• Question: What if you gave them a perfect aid?Question: What if you gave them a perfect aid?
Study 12 ProcedureStudy 12 Procedure
• Independent VariableIndependent Variable– Provision of a Perfect AidProvision of a Perfect Aid– No AidNo Aid
• For 200 slidesFor 200 slides– View SlideView Slide– Make DecisionMake Decision– View Aid’s Decision (if View Aid’s Decision (if
Perfect Aid Condition)Perfect Aid Condition)
• Key 100 trialsKey 100 trials– View SlideView Slide– View Aid’s Decision (if View Aid’s Decision (if
Perfect Condition)Perfect Condition)– View Slide Again (if want)View Slide Again (if want)– Make DecisionMake Decision– Indicate ConfidenceIndicate Confidence
0
20
40
60
80
100
Number of Relooks
Alone Perfect Aid
Nu
mb
er
of
Re
loo
ks
Study 12 ResultsStudy 12 Results
0
0.1
0.2
0.3
0.4
0.5
Target Absent (DISUSE) Target Present (DISUSE)
Alone Perfect Aid
Aid Correctly Says ABSENT
Aid Correctly SaysPRESENT
Tra
ns
form
ed p
(err
or)
Study 12 ResultsStudy 12 Results
Study 12 ImplicationsStudy 12 Implications
• Bottom LineBottom Line• People did not always rely on the perfect People did not always rely on the perfect
automated aidautomated aid• Perhaps the need to control led people to ignore Perhaps the need to control led people to ignore
some of the perfect aid’s decisionssome of the perfect aid’s decisions• QuestionQuestion• What would happen if you gave people an aid What would happen if you gave people an aid
that was certain to be WRONG?that was certain to be WRONG?
Study 13 ProcedureStudy 13 Procedure
• Automated is ALWAYS Automated is ALWAYS wrongwrong
• For 200 slidesFor 200 slides– View SlideView Slide– Make DecisionMake Decision– View Aid’s DecisionView Aid’s Decision
• Key 100 trialsKey 100 trials– View SlideView Slide– View Aid’s DecisionView Aid’s Decision– View Slide Again (if want)View Slide Again (if want)– Make DecisionMake Decision– Indicate ConfidenceIndicate Confidence
0
20
40
60
80
100
Aid Always Wrong Aid Always Right No Aid
Nu
mb
er
of
Re
loo
ks
Study 13 Results (with Study 12)Study 13 Results (with Study 12)
Study 13Study 13 Study 12Study 12
0
0.1
0.2
0.3
0.4
0.5
Target Absent (DISUSE) Target Present (DISUSE)Aid INCorrectly
Says PRESENT
Aid INCorrectly Says
ABSENT
Tra
ns
form
ed p
(err
or)
Study 13 ResultsStudy 13 Results
Study 13 ImplicationsStudy 13 Implications • Bottom LineBottom Line• People did not always ignore an automated aid People did not always ignore an automated aid
known to be wrong on every trialknown to be wrong on every trial• Therefore, the need for control is not the cause of the Therefore, the need for control is not the cause of the
inappropriate resistance of the perfect aidinappropriate resistance of the perfect aid• Perhaps the experience with the aid led participants to Perhaps the experience with the aid led participants to
distrust the aiddistrust the aid• Question: What would happen if you gave people a perfect Question: What would happen if you gave people a perfect
automated aid but did not give them much experience with automated aid but did not give them much experience with the aid?the aid?
Study 14 ProcedureStudy 14 Procedure
• All participants are told their automated All participants are told their automated aid is perfectaid is perfect
• Key 100 trialsKey 100 trials– View SlideView Slide– View Aid’s Decision View Aid’s Decision – View Slide Again (if want)View Slide Again (if want)– Make DecisionMake Decision– Indicate ConfidenceIndicate Confidence
0
20
40
60
80
100
Wrong AidExperience
Correct AidExperience
Correct AidNo
Experience
No Aid
Nu
mb
er
of
Re
loo
ks
Study 14 Results (with Studies 12&13)Study 14 Results (with Studies 12&13)
Study 13 Study 12 Study 14 Study 12Study 13 Study 12 Study 14 Study 12
0
0.1
0.2
0.3
0.4
0.5
Target Absent (DISUSE) Target Present (DISUSE)Aid Correctly Says
ABSENTAid Correctly Says
PRESENT
Tra
ns
form
ed p
(err
or)
Study 14 ResultsStudy 14 Results
Study 14 ImplicationsStudy 14 Implications • Bottom LineBottom Line• People did not always rely on a perfect People did not always rely on a perfect
automated aid—even without experience with automated aid—even without experience with the aidthe aid
• Removing experience with the perfect aid Removing experience with the perfect aid slightly decreased the number of re-looks, but it slightly decreased the number of re-looks, but it also decreased reliance on the perfect aidalso decreased reliance on the perfect aid
• Question: What if the automated aid was presented as Question: What if the automated aid was presented as an expert?an expert?
Study 15 DesignStudy 15 Design
• Independent VariablesIndependent Variables– Presumed Expertise of the Aid (between)Presumed Expertise of the Aid (between)– Type of Aid (within)Type of Aid (within)
• HumanHuman• AutomatedAutomated
• Dependent VariablesDependent Variables– Transformed p(error |aid incorrect) = MISUSETransformed p(error |aid incorrect) = MISUSE– Transformed p(error|aid correct) = DISUSETransformed p(error|aid correct) = DISUSE
Study 15 ProcedureStudy 15 Procedure
• 100 trials100 trials– View SlideView Slide– ChooseChoose
• View Automated Aid’s DecisionView Automated Aid’s Decision• View Human Aid’s DecisionView Human Aid’s Decision• Make DecisionMake Decision
– View Aid’s Decision (if requested) View Aid’s Decision (if requested) – Make DecisionMake Decision– Indicate ConfidenceIndicate Confidence
AnalysesAnalyses• 2 (Expert vs Not) x 3 (Aid: None vs Human vs
Auto) ANOVAs were performed for errors made when target was present and absent.
• When target was absent– Main effect for Aid: Participants made more errors if
they did not ask for aid’s help than if they did
• When target was present:– Main effect for Aid: Participants made more errors if
they did not ask for the aid’s help than if they did– Main effect for Expertise of Aid: Participants with
expert aids made fewer errors
0
0.2
0.4
0.6
0.8
1
No Aid Human Aid Automated Aid
p(error|aid error) p(error|aid correct)
Study 15 ResultsStudy 15 Results
Study 15 ResultsStudy 15 Results
• Examination of the number of trials participants asked to view the decision made by the automated aid, human aid, or no aid revealed some individual differences– 16 participants did not view an aid’s decision for
over half of the trials; 4 of these NEVER viewed an aid’s decision
– 6 participants viewed the human aid’s decision for over half the trials
– Only one participant viewed the automated aid’s decision for over half the trials
Study 15Study 15
Please describe the usefulness of the Please describe the usefulness of the automated/human aid. Pretend you automated/human aid. Pretend you are writing to the next participant are writing to the next participant coming into this experiment. Give the coming into this experiment. Give the next person some advice concerning next person some advice concerning this automated/human aid.this automated/human aid.
ResultsResults
• Can linguistic analysis identify the people who are likely to ignore an aid or prefer a human aid over an automated aid?
Category Category (Examples) (Examples)• pronouns pronouns (I, me, we, you) (I, me, we, you) • positive emotions positive emotions (happy, pride, good)(happy, pride, good)• negative emotions negative emotions (hate, afraid, sad)(hate, afraid, sad)• insight insight (think, know, consider)(think, know, consider)• time time (past, present, future)(past, present, future)• communication communication (talk, share, converse)(talk, share, converse)• anxiety words anxiety words (nervous, afraid, tense)(nervous, afraid, tense)
LIWC analyzes text word-by-word and categorizes LIWC analyzes text word-by-word and categorizes the text into 74 different linguistic dimensionsthe text into 74 different linguistic dimensions
LIWC VariablesLIWC Variables
0
2
4
6
8
10
12
Article Use Inclusive Words
Human Automated
People Used Less Complex Language People Used Less Complex Language When Writing about an Automated Aid When Writing about an Automated Aid
than a Human Aidthan a Human Aid
0
2
4
6
8
10
12
Causal Words Social Words
Human Automated
Operators Used More Cognitive and Social Operators Used More Cognitive and Social Processes When Writing about the Human Processes When Writing about the Human
than Automated Aidthan Automated Aid
0
0.5
1
1.5
2
2.5
3
Second Person Pronouns References to Other People
Human Automated
Pronoun Use Varied with Type of AidPronoun Use Varied with Type of Aid
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Anxiety Words Anger
Human Automated
Negative Emotions Were Expressed More Negative Emotions Were Expressed More When Discussing the Automated than When Discussing the Automated than
Human AidHuman Aid
The more participants followed the human aid into The more participants followed the human aid into an error (i.e., misused the human aid):an error (i.e., misused the human aid):
• The more they described the automated aid with– Discrepancy words
– Future tense
• The more they described the human aid with – Negate words
– Fewer cognitive mechanism words
• The bigger the difference between auto and human aid descriptions for– Negate words
– Discrepancy words
The more participants followed the non-expert The more participants followed the non-expert human aid into an error (i.e., misused the human human aid into an error (i.e., misused the human
aid):aid):• The more they described the automated aid with– Discrepancy words
• The more they described the human aid with – Communication words– Negate words– Fewer affect words
• The bigger the difference between auto and human aid descriptions for– Discrepancy words – Negate words – First person pronouns (human > auto)– First person singular pronouns (human > auto)
Study 15 ImplicationsStudy 15 Implications • Bottom LineBottom Line• People show a preference for self-reliancePeople show a preference for self-reliance• People were more likely to view decisions People were more likely to view decisions
of human rather than automated aidsof human rather than automated aids• People were equally likely to rely on human People were equally likely to rely on human
and automated aidsand automated aids• Linguistic analysis was able to detect Linguistic analysis was able to detect
differences in people’s conceptions of differences in people’s conceptions of human and automated aidshuman and automated aids
ConclusionsConclusions
–Supplying automated systems will not always Supplying automated systems will not always improve performanceimprove performance
–Predicting automation usage decisions is Predicting automation usage decisions is complex even with a simple taskcomplex even with a simple task
Future ResearchFuture Research
– Replicating the results in a combat-like environmentReplicating the results in a combat-like environment
– Examining the effect of trainingExamining the effect of training
– Further testing of the modelFurther testing of the model– Determine Effect SizesDetermine Effect Sizes– Examine PathwaysExamine Pathways
– Applying results to system developmentApplying results to system development
What We Know for System DevelopmentWhat We Know for System Development• Sometimes, people provided with an automated decision aid Sometimes, people provided with an automated decision aid
performed WORSE than those without itperformed WORSE than those without it• Improving the reliability of the automated decision aid did NOT Improving the reliability of the automated decision aid did NOT
improve the performance of the human-automation teamimprove the performance of the human-automation team• Training in conditions in which the aid will and will not be reliable Training in conditions in which the aid will and will not be reliable
was not effective in promoting appropriate automation usewas not effective in promoting appropriate automation use• However, information as to why the automated aid might err led However, information as to why the automated aid might err led
people to trust the aid more and appropriately rely on the aidpeople to trust the aid more and appropriately rely on the aid
In addition… these techniques may lead to more In addition… these techniques may lead to more appropriate automation relianceappropriate automation reliance
• Reduce people’s initial trust in the automated aidReduce people’s initial trust in the automated aid• Provide participants with various types of Provide participants with various types of
information concerning their and their automated information concerning their and their automated aid’s performanceaid’s performance
• Provide participants with continuous feedback and Provide participants with continuous feedback and eliminate the possibility of the detection of obvious eliminate the possibility of the detection of obvious errorserrors
Questions??Questions??