Computational models of human visual attention driven by auditory cues

Akisato Kimura, Ph.D

NTT Communication Science Laboratories

(Most of the content presented in this talk are based on the collaborative research with National Institute of Informatics, Japan.)

Visual attention

Visual attention is a built-in mechanism of the human visual system for scene understanding.

http://www.tobii.com/eye-tracking-research/global/library/white-papers/tobii-eye-tracking-white-paper/

Simulating visual attention is essential

Such a pre-selection mechanism would be essential in enabling computers to undertake

[http://www.icub.org]

Visual assistance

[https://www.google.com/glass]

Object detection

[Donoser et al. 09]

Saliency as a measure of attention

Saliency = attractiveness of visual attention

• Simple, easy to implement, reasonable outputs

Input image Saliency map [Itti et al. 98]

Estimating human visual focus of attention

Related work

Visual saliency

• Saliency map model [Itti 1998]

• Shannon self-information [Bruce 2005]

• Incorporating temporal dynamics [Itti 2009]

[Itti et al. 98] [Bruce et al. 05]

Input image

Visual attention modulated by audios

Sounds are strongly related to events that draw human visual attention.

Without audio

With audio

[Song et al.11]

Speaking

Related work

Visual saliency

• Saliency map model [Itti 1998]

• Shannon self information [Bruce 2005]

• Incorporating temporal dynamics [Itti 2009]

Auditory saliency

• Center-surround mechanism [Kayser 2005]

• Bayesian surprise [Schauerte 2013]

Audio-visual saliency

• Multi-modal saliency for robotics []

• Sound source localization[Nakajima 2013]

[Itti et al. 98] [Bruce et al. 05]

[Kayser et al. 05]Audio spectrogram

Input image

[Itti et al. 03] [Nakajima et al. 13]Input video

Human visual attention models with the help of auditory information is underway.

Main content of this talk

Our recent challenges to simulate human visual attention driven by auditory cues

• Auditory information plays a supportive rolein contrast to standard multi-modal fusion approaches

• Our strategy is built on two psychophysical findings

1. Audio-visual temporal alignment leads to benefits changes are both synchronized and transient[Van der Burget, PLoS One 2010]

2. Auditory attention modulates visual attention in a feature-specific manner [Aveninen, PLoS ONE 2012]

Our strategy

1. Audio-visual temporal alignment leads to benefits changes are both synchronized and transient[Van der Burget, PLoS One 2010]

2. Auditory attention modulates visual attention in a feature-specific manner [Aveninen, PLoS ONE 2012]

Following those findings…

1. detect transient events in visual and auditory domains separately

2. look for visual features synchronized withdetected auditory events

3. modulate saliency maps by feature selection.

Previous method – Bayesian surprise

Image signal

Input video

Bayesian surprise

Visual saliency(from only visual features)

Conventional saliency maps

Our strategy

Audio signalImage signal

Input video

Bayesian surprise

Visual saliency(from selected visual features)

Auditory Surprise

Selecting visual features synchronized with the auditory events

Modulating saliency mapswith the selected features

Proposed saliency maps

Bayesian surprise

Audio signalImage signal

Input video

Bayesian surprise

Auditory Surprise

Concept of Bayesian surprise

Continuously similar features Low saliency values

Unexpected features high saliency values

Visual Bayesian surprise

Intensity×6Color×12Orientation×24Flicker×6Motion×24

Kullback-Leibler divergence

Observations = Feature maps

Input video

Gaussian pyramid scale

surprise

Visual surprise for 72 feature maps

Input video Visual surprise

PosteriorBayes

72 visual feature maps

[Itti, Vision Research 2009]

Auditory Bayesian surprise

Spectrograms as an observation

Audio signal

Spectrogram

Auditory surprise

Prior Posterior

Observation at frequency 𝜔

Surprise at frequency 𝜔

Averaged over frequencies

Auditory surprise

[Schauerte, ICASSP2013]

Audio-visual synchronization

Bayesian surprise

Auditory Surprise

Selecting visual featuressynchronized with the audio

Correlation-based detection

Type of features

The window width depends on the length of auditory events

𝜃𝑠

Visual surprise Auditory surprise

Feature 𝑓

Averaging over pixels

Auditory event

360 features

Calculating correlation

Visual feature selection

Bayesian surprise

Auditory Surprise

Selecting visual features synchronized with the audio

Modulating saliency mapswith the selected features

Proposed saliency maps

Selecting visual features

Type of features Binarization

Frequency of “synchronization”

Selected features

Voting with threshold 𝜃𝑐

TimeFinal saliency map

Emphasizing selected features by summing up only selected features

360 types of features𝑁 < 360 types of features

Experimental setup

Detecting scan-paths for ground truth• 15 subjects

• 6 videos (The DIEM project)

• Using Tobii TX300

Evaluation criteria

• Normalized Scanpath Saliency (NSS) [Peters 2009]

• Baseline:• Saliency map model [Itti 2003], Bayesian surprise [Itti 2009],

Sound source localization [Nakajima 2013]

Experimental results – summary

The proposed model produced best NSS scores for all the videos

Qualitative evaluation – Video 2

Input Baseline

Auditory surprise Proposed

Detailed evaluation – Video 1

NSS(Proposed)

NSS(Baseline)

Auditory surprise

Auditory event

Feature Intensity Color Orientation Flicker Motion Total

Baseline 30 60 120 30 120 360

Proposed 8 17 46 0 0 71

Selected visual features

The proposed model outperformed the baseline in many frames

Some extensions

Drawbacks of the proposed method

• 2-pass algorithm:Whole the video should be scanned first to detect synchronization.

Recent updates

• Sequential estimation of visual & auditory surprise via exponential smoothing

Video 1 Video 2 Video 3 Video 4 Video 5 Video 6

Itti2009 2.896 1.816 0.790 1.209 0.318 0.513

Nakajima2013 1.857 0.992 0.540 1.073 0.368 0.216

Proposed (new) 3.077 1.820 0.791 1.273 0.318 0.513

Conclusion

Our recent challenges to simulate human visual attention driven by auditory cues

• Auditory information plays a supportive role

• Our model is built on recent psychophysical findings

Human visual attention models with the help of auditory information is underway.

• Auditory attention models

• Auditory cues other than synchronization

Reference

• Kimura, Yonetani, Hirayama “Computational models of human visual attention and their implementations: A survey,” IEICE Transactions on Information and Systems, Vol.E96-D, No.3, 2013.

• Nakajima, Sugimoto, Kawamoto “Incorporating audio signals into constructing a visual saliency map,” Proc. Pacific-Rim Symposium on Image and Video Technology (PSIVT2013).

• Nakajima, Kimura, Sugimoto, Kashino “Visual attention driven by auditory cues: Selecting visual features in synchronization with attracting auditory events,” Proc. International Conference on Multimedia Modeling (MMM2015).

• Nakajima, Kimura, Sugimoto, Kashino “An online computational model of human visual attention considering spatio-temporal synchronization with auditory events,” IPSJ Technical Report, CVIM195-57, 2015 (in Japanese).

Computational models of human visual attention driven by auditory cues

Technology

Auditory Nerve Representation of a Complex Communication ...webpage.pace.edu/jschwartz2/schwartz_audiotory_nerve... · pranica, 1965; Rose and Capranica, 1984). Temporal cues are

I Smell Creativity: Exploring the Effects of Olfactory and Auditory … · 2020. 9. 14. · I Smell Creativity: Exploring the Effects of Olfactory and Auditory Cues to Support Creative

A THEORY AND COMPUTATIONAL MODEL OF ... A THEORY AND COMPUTATIONAL MODEL_ OF AUDITORY MONAURAL SOUND SEPARATION , A DISSERTATION SUBMITTED TO THE DEPARTMENT OF ELECTRICAL ENGINEERING

Computational modelling suggests that temporal integration ... · Computational modelling suggests that temporal integration results from synaptic adaptation in auditory cortex Patrick

Vertical Illusory Self-motion Through Haptic Stimulation of …...as illusions induced via auditory feedback including explicit motion cues, i.e., moving sound ﬁelds. Riecke et al

A behaviorally inspired fusion approach for computational ...cvsp.cs.ntua.gr/publications/jpubl+bchap/TsiamiEtAl_BehaviorInspired... · Although visual and auditory stimuli often

Auditory Forebrain Auditory Thalamus Auditory Cortex Auditory Projections to Polysensory Cortex Plasticity

Fine Arts & Creativity · Visit past newsletter issues 2012-13 #7 & #8 to learn more about DL and “Informational Text.” WORDS, #, Images, Auditory Cues Disciplinary Literacy Arts

Impoverished auditory cues limit engagement of brain networks … · 2019. 4. 22. · Impoverished auditory cues limit engagement of brain networks controlling spatial selective attention

Audiovisual Speech Perception with Degraded Auditory Cues

Aural servo: sensor-based control from robot audition · knowledge in psycho-acoustics and physiology for extracting auditory cues from an articial hearing system. Auditory cues such

Temporal Auditory Processing in Infancy · listeners can use absolute cues to guide performance. When the patterns repeat with variations, however, listeners must use relational cues

An Initial Exploration of a Multi-Sensory Design Space ...gogo/papers/3dui2016_Feng.pdfand multi-sensory cues (e.g., auditory and tactile cues). While the first two have been fairly

Prediction-driven computational auditory scene …...3 Prediction-driven computational auditory scene analysis by Daniel P. W. Ellis Submitted to the Department of Electrical Engineering

Rhythmic auditory cues shape neural network recruitment in ... › 2018 › 12 › ... · Rhythmic auditory cues shape neural network recruitment in Parkinson’s disease during repetitive

16 Separation of Speech by Computational Auditory Scene ...web.cse.ohio-state.edu/~wang.77/papers/Brown-Wang05.pdf · Abstract. The term auditory scene analysis (ASA) refers to the

Weighting of acoustic cues shifts to frication duration in identification of fricatives/affricates when auditory properties are degraded due to aging

Auditory Scene Analysis: phenomena, theories and computational

Computational Modeling of the Auditory Periphery: From Soundfiles to Spiketrains Marcos Cantu Center for Computational Neuroscience and Neural Technology

An Auditory Output Brain–Computer Interface for Speech ......auditory output approach employing a motor-speech theoretical perspective, draw ing from computational modeling of the