CROSSMODAL EFFECTS IN TACTILE AND VISUAL

SIGNALING

Can we get the message through?

Sensing

Communicating

Effective action depends on…

Presenting

Processing

Understanding the situation

Picture of thermometer in

HMMWV taken 31 May, ‘04

in central Baghdad.

Sensing

Sensing… with the senses covered…

Effective action depends on

Communicating

Communicating… with the senses in extreme context

http://www.globalsecurity.org/military/systems/ground/images/xuv_pic3.jpg

TOC - ABCS

GPS

Full FBCB2 onLower Tactical

Internet

L-Band

CO/PLT Leaders/Sgts

CO/PLT Voice Nets

SQD Leader

TM Leaders

SQD Leader

TM Leaders

SQD Leader

Squad Voice Net

Squad Voice Net

Squad Voice Net

BFT (FBCB2)

GPSBattalion SA NetDBCS (T)

DBCS (P)

Who else needs to know?

• Gilson & Fenton (1974) A tactile display presenting aircraft control information was comparable to using a visual display and in some cases actually produced a reduction in tracking error, altitude variations decreased as well as speed variations.

• Zlontik (1988) The use of tactile signaling helped pilots and air traffic controllers overcome some of their challenges.

• Gilland & Schlegel (1994) reported that tactile communication applied to the head lowered performance on a concurrent task.

• Sklar & Sarter (1999) Showed tactile signals are able to be perceived concurrently with auditory or visual information.

• Cholewiak, Brill, & Schwab (2004) found that for the torso, a ring of eight loci of vibration is the most one can resolve with accuracy exceeding 90%.

• Hoop, Smith, Clegg & Heggestad (2005) Tactile signals aided in interruption management.

• Pettitt, Redden, & Carstens (2006) found that tactile presented arm and hand signals could be interpreted faster than standard visually presented signals when undergoing dynamic movement tasks.

Literature Review

Tactile Patterns emulating Standard Army Hand Signals* (FM 21-60)

Tactons*…

Merlo, J.L., Stafford S.C., Gilson, R. & Hancock, P.A. (2006). The effects of physiological stress on tactile communication. Proceedings of the Human Factors and Ergonomics Society 50th Annual Meeting, San Francisco, CA.

Merlo, J.L., Terrence, P.I. Stafford S.C., Gilson, R. & Hancock, P.A., Redden E.S., Krausman A., Pettitt, R., White, T.L., & Carstens, C.B. (2006). The effects of dynamic environments and physiological stress on tactile communication and signaling. Proceedings of the Society for Human Performance in Extreme Environments 4th Annual Meeting, San Francisco, CA.

Merlo, J.L., Terrence, P.I. Stafford S.C., Gilson, R. & Hancock, P.A., Redden, E.S., Krausman, A., Pettitt, R., White, T.L., & Carstens, C.B. (2006). Communicating through the use of vibrotactile displays for dismounted and mounted soldiers. Proceedings of the Twenty-fifth Army Science Conference, Orlando, FL.

Multi-Modal Benefits• Spence and Driver (2004) Attention and then perception is largely a multi-

sensory construction

• Stein and Meredith (1993) bi-modal and tri-modal neurons have stronger cellular response with two sensory modalities than with uni-modal stimulation

• Wallace, Meredith, Stein (1998) found that combinations of two different sensory stimuli have been shown to significantly enhance the response of Superior Colliculus neurons above those evoked by either uni-modal stimulus

• Allman and Meredith (2007) found uni-modal visual neurons had their visual

response modulated by concurrently presented auditory stimuli

• Current research has explored these conflicts of congruent and incongruent cross-modal paradigms in many of the possible permutations and combinations of very basic visual, auditory, kinesthetic and tactile stimuli (Soto-Faraco, Lyons, Gazzaniga, Spence, & Kingstone, 2002; Soto-Faraco, Morein-Zamir, & Kingstone, 2005; Soto-Faraco, Spence, & Kingstone, 2004a; Spence & Driver, 1997; Spence & Walton, 2005).

Experimental Method

• visual only (video presentation of the signals)

• tactile only (tactile presentation of the signals)

• both visual and tactile simultaneous and congruent (i.e. the same signals were presented both visually on the video and through the tactile system)

• both visual and tactile simultaneous and incongruent (i.e. the visually presented signal did not match the presented tactile signal).

Equipment

• an acoustic transducers

• displace 200-300 Hz sinusoidal vibrations onto the skin

• C2’s contactor is 7 mm, with a 1 mm gap separating it from the tactor aluminum housing

• 17 grams

• 8 tactors mounted on an elastic belt with acoustic padding

• 8 X 45 degrees = 360 degrees

• Tactor Control Box (TCB) Stores tactile messages and contains bluetooth receiver

• Total system under 2 lbs

Spatial layout of Tactile Display

1FRONT

3

2

RIGHT

54

BACK

LEFT7

6

8

N

NE

E

SE

SSW

W

NW

Arm/ Hand Signal

Tactile Pattern Visual Pattern

Attention

Sequenced activation of three forward tactors creating a linear motion back and forth across the front of the body

Halt Four tactors simultaneously actuated on the sides of the body

Rally Sequenced activation of all tactors creating a circular motion around the body

Move Out

Sequenced back-to-front activation of tactors creating movement from each side of the body which converges in the front

NBCSequenced activation on both sides simultaneously creating three distinct impulses on the sides of the body

Method

Attention

RALLY

RALLY

Equipment

• Samsung Q1 Ultra Mobile Intel Celeron M ULV (900 MHz) 7” WVGA (800 x 480) LCD

SoftwareHardware

• Sound dampening earmuffs reduction rating of 11.3 decibels at 250 Hz

• Windows XP, tablet• LabVIEW 8.2 National Instruments• Microsoft Office 2003 • SPSS 11.5• Windows Media Player

Experimental Procedure

20 Participants

• 10 Males

• 10 Females

• Mode age 19 years

Training session

• Learned both visual and tactile signals simultaneously

• Advanced only after mastery

• Lasted approximately 10 minutes

Experimental block• 10 trials visual only • 10 trials tactile only• 20 congruent trials• 20 incongruent trials 60 total trials per block

2 x Blocks = 120 trials

Signals

• Attention

• Halt

• Move Out

• NBC

• Rally

Screen Capture from Visual interface

Participants chose the appropriate signal as fast a possible after signal presentation. The tactile only presentation showed a video of the soldier standing at the rigid position attention.

0

500

1000

1500

2000

2500

3000

Concurrent Visual Only Tactile Only

Condition

Mea

n R

espo

nse

Tim

e (m

s)

Initial Results

Response Time x Condition

t(19)=-2.25, p≤.04 t(19)=-2.16, p≤.04

t(19)=-3.98, p≤.01

Oneway ANOVA

( F(1, 19)=473.445, p<.000, partial eta squared = .961, observed power, β= 1.000 )

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%

Concurrent Visual Tactile

Condition

Per

cen

t C

orr

ect

Accuracy x Condition

t(19)=2.00, p≤.06

t(19)=4.03, p≤.01

t(19)=1.27, p≤.22

Initial Results

0

500

1000

1500

2000

2500

3000

Concurrent and congruent Visual Only Tactile Only Response not matchingeither presentation

Condition

Me

an

Re

sp

on

se

Tim

e (

ms

)

Initial Results

0

5

10

15

20

25

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Partcipant

Fre

qu

ency

of

Mo

dal

ity

Sel

ecti

on

Tactile

Visual

0

5

10

15

20

25

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Partcipant

Fre

qu

ency

of

Mo

dal

ity

Sel

ecti

on

Tactile

VisualBlo

ck 2

Blo

ck 1

Discussion

• Data should provide valuable information on:– Congruency effects– Selected attention– Interference effects– Tactile signal efficiency– Modality interactions

• Future research for tactile signaling includes:– Spatial references– Temporal importance– Alternate inputs

Questions?