Pixels to Voxels: Modelling Visual Representations in the Human Brain

Pulkit Agrawal, Dustin Stansbury, Jitendra Malik, Jack Gallant

(Under Review, arXiv preprint available online)

Thanks to Brian Cheung, Mark Lescroart and Alex Huth for the helpful discussions

Goal

“ Obtain a functional mapping of the human visual cortex ”

or

“ What visual features does the brain code for? ”

Why solve this?

- We can’t fix it, unless we know what it does → Neural Prosthesis.

- Brain Reading.

- A scientific pursuit. “What are the underlying computational principles used by biological vision?”

- Understanding limitations of state of art computer vision systems.

functional Magnetic Resonance Imaging (fMRI)

What does fMRI measure?

Neural Activity → Oxygen Consumed → Oxygen level in blood drops

fMRI measures “local” changes in Oxygen level

Indirect measurement of neural activity!

[Slide taken from Tom Mitchell]

http://www.youtube.com/watch?v=LRmNjJ8D_Bo

Traditional Neuroscience approach

- Hypothesis Driven. - Experimenter’s bias.

- Measures binary contrast only!

- Cannot provide holistic description of stimulus tuning of voxels.

Basic Idea

[Gallant et al.]

Basic Idea

[Gallant et al.]

Gabor Model → Early Visual Cortex

[Kay et al.]

Semantic Model → Higher Visual Cortex

[Stansburry et al.]

i.e. until now,

- Different models for different brain areas

- Human Annotation for semantic Model :(

- No model explains mid-level visual areas :(

[Stansburry et al.]

This work,

- Modelling the entire visual hierarchy

- No need for human annotation

- Enable exploratory analysis of neural representations!!

Overview

ConvNet v/s Semantic Model

Semantic Model

Con

vNet

Mod

el

Functional Selectivity

Increased Activity Decreased Activity

Clustering in EBA

Increased Activity Decreased Activity

In a nutshell

- A single annotation free for modelling brain responses throughout the visual cortex.

- Replicate results of multiple localizer studies in a single experiment.

- Novel insights into representations such as clustering in EBA.

Questions?

Fisher Vector v/s Semantic Model

Semantic Model

Fish

er V

ecto

r M

odel

ConvNet v/s Fisher Vector

Fisher Vector Model

Con

vNet

Mod

el

Brief Description of Fisher Vectors