Temporal and Cross-Subject Probabilistic Models for fMRI Prediction Tasks
Alexis BattleGal ChechikDaphne Koller
Department of Computer ScienceStanford University
PBAI Competition
● Provided rich data set– Interesting interactions across time, subjects, and
stimuli
● Challenged us to come up with reliable techniques
● Thanks to the organizers!
Key Points
● Predictive voxels selected from whole brain
● Probabilistic model makes use of additional correlations– Subjects’ ratings across time steps
– Ratings between subjects
● Learn strength of each relationship
Modeling the fMRI Domain
A joint distributionin high dimension
Voxels across time
BOLD signal
Ratings across time
User Ratings
funnytimbodylanguage
Modeling the fMRI Domain
A joint distributionin high dimension
Voxels across time
BOLD signal
Ratings across time
User Ratings
funnytimbodylanguage
Training:Use two movies to learn the relations between voxels and ratings
Modeling the fMRI Domain
A joint distributionin high dimension
Voxels across time
BOLD signal
Ratings across time
User Ratings
funnytimbodylanguage
Training:Use two movies to learn the relations between voxels and ratings
Testing:Use the learned relations to predict ratings from fMRImeasurements
Probabilistic Model
• Each voxel measurement• Each rating to predict
Language
fromVox1 Vox2 Vox3
Probabilistic Model
• Each voxel measurement• Each rating to predict• Rating predicted from voxel measurements
– Linear regression model (Gaussian distribution)
Language
fromVox1 Vox2 Vox3
Probabilistic Model
• Each voxel measurement• Each rating to predict• Rating predicted from voxel measurements
– Linear regression model (Gaussian distribution)
Language
fromVox1 Vox2 Vox3• Selected predictive
voxels from whole brain• Regularize (Ridge,
Lasso) to handle noise
Probabilistic Model
Language
Vox1 Vox2
Language
Vox1 Vox2
…T =1 T =2
Probabilistic Model
● Ratings correlated across time– Language at time 1 makes language at time 2 likely
Language
Vox1 Vox2
Language
Vox1 Vox2
…T =1 T =2
Probabilistic Model
● Ratings correlated across time– Language at time 1 makes language at time 2 likely
Language
Vox1 Vox2
Language
Vox1 Vox2
…T =1 T =2
Probabilistic Model
● Ratings correlated across time– Language at time 1 makes language at time 2 likely
– Weight A – how correlated?
Language
Vox1 Vox2
Language
Vox1 Vox2
…T =1 T =2
A*Lang (1)*Lang(2)
A
Probabilistic Model
Subject 1
Subject 2
Language
Vox1 Vox2
Language
Vox1 Vox2
T =1 T =2
Language
Vox1 Vox2
Language
Vox1 Vox2
…
Probabilistic Model
● Ratings likely to be correlated between subjects
Subject 1
Subject 2
Language
Vox1 Vox2
Language
Vox1 Vox2
T =1 T =2
Language
Vox1 Vox2
Language
Vox1 Vox2
…
Probabilistic Model
● Ratings likely to be correlated between subjects– Weighted correlation, NOT equality
Subject 1
Subject 2
Language
Vox1 Vox2
Language
Vox1 Vox2
T =1 T =2
Language
Vox1 Vox2
Language
Vox1 Vox2
…
B B
Probabilistic Model
Joint model over all time points:
Time
…Sub1
Sub2
Gaussian Markov Random Field – joint Gaussian over all rating nodes conditioned on voxel data
Voxel Parameters
● Regularized linear regression for voxel parameters
Language
Vox1 Vox2 Vox3
Voxel Parameters
● Regularized linear regression for voxel parameters
Language
Vox1 Vox2 Vox3
Voxel Parameters
● Regularized linear regression for voxel parameters
β1 = 0.45
β2 = 0.55
Language
Vox1 Vox2 Vox3
Inter-Rating parameters
L(1)
Vox1 Vox2
L(2)
Vox1 Vox2B
● Other weights also learned from data– Example: cross-subject weights
Inter-Rating parameters
L(1)
Vox1 Vox2
L(2)
Vox1 Vox2B
● Other weights also learned from data– Example: cross-subject weights
Attention Faces
Inter-Rating parameters
L(1)
Vox1 Vox2
L(2)
Vox1 Vox2B
B = 0.3 B = 0.7
● Other weights also learned from data– Example: cross-subject weights
Attention Faces
Prediction Results
● Use full learned model, including all weights
● Predict ratings for a new movie given fMRI data
Prediction Results
● Use full learned model, including all weights
● Predict ratings for a new movie given fMRI data
Prediction Results
● Comparison to models without time or subject interactions
Voxel Selection
● Voxels selected by correlation with rating– Number of voxels determined by cross-validation
Voxel Selection
● Voxels selected by correlation with rating– Number of voxels determined by cross-validation
Selected Voxels
Faces Language
L L
Selected Voxels
Motion Arousal
L L
Voxel Selection
● Voxels selected for Language included some in ‘Face’ regions:
L
Voxel Selection
● Voxels selected for Language included some in ‘Face’ regions:
L
● Language and face stimuli correlated in videos
● Complex, interwoven stimuli affect voxel specificity
Voxel Selection
0.330.38
● Voxel selection extension – “spatial bias”– Prefer grouped voxels
* after competition submission
Voxel Selection
* after competition submission
0.330.38
● Voxel selection extension – “spatial bias”– Prefer grouped voxels
Voxel Selection
* after competition submission
0.330.38
● Voxel selection extension – “spatial bias”– Prefer grouped voxels
– Additional terms in linear regression objective:
● |β1| |β2| D(Vox1, Vox2)
D
|| Vox1 –Vox2||2
Adding Spatial Bias
Faces
L L
Conclusions
● Reliable prediction of subjective ratings from fMRI data
● Time step correlations aid in prediction reliability
● Cross-subject correlations also beneficial
● Individual voxels selected from whole brain– Reliability from regularization
– Some found in expected regions
– Some “cross-over” for correlated prediction tasks
Recommended
