Detection of Affective States in Human-Computer InteractionYing Gao

“Affective Sensing” needed for Affective Computing

Aim: To give computers the capability of reacting appropriately to their users’ emotional states.

COMPUTER

USER

User’s Affective

Model

Computer’s Affective

Model

AffectiveSensing

AffectExpression

Pre-Programmed Interplay Example Uses: Intelligent Tutoring System - Fatigue recognition

Student Engagement in ResearchArmando Barreto, Ph.D.

First Example:

GSR, BVP, PD

time (seconds)

volta

ge (v

olts

)GSR respnse

Stimulus

LatencyAmplitude

RiseTime

HalfRecoveryTime

Time (seconds)

Vol

tage

(vol

ts)

Amplitude

Period (IBI)

PupilPupil

GSR

BVP

PD

Affective Sensing through Processing of Physiological Signals

Framework:

Affective Sensing

Interaction Loop

Physiological Signal

MonitoringPD

GSR

BVP

Affective Detection through Signal

Processing MethodFeature

Selection and

Extraction

Affective Recognition through Machine Learning AlgorithmSVM

ROC

Affective

Controller

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

false positive rate

true

posi

tive

rate

ROC Curve of PMPDmean Signal

PMPDmean ROC curve (AUROC = 0.9331)

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

False Positive Rate

True

Pos

itive

Rat

e

ROC and ROCCH Curve for Feature BVPl2h

BVPL2H ROC curve (AUROC = 0.5432)

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

false positive rate

true

posi

tive

rate

ROC Curve of GSRmean Signal

GSRmean ROC curve (AUROC = 0.6780)

3D Sound for Human-Computer Interaction: Customization & Effective Areas

Kenneth John Faller

• Problem 1 – HRTFs vary from person to person . Must measure for each user or “customize”

• Problem 2 – Human accuracy in localizing sound source placement (real or virtual) varies around the listener What are the regions of maximum accuracy?

“Real” HRTFs

FIR

FIR

FIR

FIR

1.wav 2.wav

Emulated HRTFs

Digital filters

Second Example:

1 - Customization1-A: One approach to customization

requires DECOMPOSITION of the impulse response of the HRTFs (or “HRIRs”) into 2nd order functions that can be determined by physical ear measurements

1-B: Customization achieved by High-Order Singular Value Decomposition (HOSVD) Tensor Model

2- Identification of most accurate regions (elevation ranges)

Explored accuracy at +/- 90o

azimuth and different elevations

•(a) “Generic HRIRs provided the lowest performance)

•(b) Elevations between -18º and 18º yield the best accuracy

•(c) Localization accuracy degrades differently for different types of HRIRs. Generic HRIRs suffered the worst degradation.

CONCLUSIONS:

The FIU CREST Students

Dr. Ying Gao• (PhD FIU EE – Fall 2009)• Currently Visiting Instructor at the

Electrical & Computer Engineering Department, FIU

Dr. Kenneth J. Faller• (PhD FIU EE – Summer 2009)• Currently Post-doctoral Fellow at the

Structural Acoustics Branch (StAB) of NASA Langley Research Center

1 2