Hidden Process Models:Decoding Overlapping Cognitive

States with Unknown Timing

Rebecca A. HutchinsonTom M. Mitchell

Carnegie Mellon University

NIPS Workshops: New Directions on Decoding Mental States from fMRI Data

December 8, 2006

2

Overview

• Open questions we address:– Treating fMRI as the time series that it is.– Allowing the testing of hypotheses.

• Open questions we do NOT address:– Interpretability of time series or spatial representation

of activity.

• This talk– Motivation– HPMs (in 1 slide!)– Preliminary results

3

Motivation

• Goal: connect fMRI to cognitive modeling.

• Cognitive Model:– Set of cognitive processes hypothesized to occur

during a given fMRI experiment.

• Cognitive Process:– Spatial-temporal hemodynamic response function.– Timing distribution relative to experiment landmarks

(like stimulus presentations and behavioral data).

4

Study: Pictures and Sentences

• Task: Decide whether sentence describes picture correctly, indicate with button press.

• 13 normal subjects, 40 trials per subject.• Sentences and pictures describe 3 symbols: *,

+, and $, using ‘above’, ‘below’, ‘not above’, ‘not below’.

• Images are acquired every 0.5 seconds.

Read Sentence

View Picture Read Sentence

View PictureFixation

Press Button

4 sec. 8 sec.t=0

Rest

5

One Cognitive Model

Read Sentence

View Picture Read Sentence

View PictureFixation

Press Button

4 sec. 8 sec.t=0

Rest

• ViewPicture – begins when picture stimulus is presented

• ReadSentence– begins when sentence stimulus is presented

• Decide– begins within 4 seconds of 2nd stimulus

ViewPicture or ReadSentence ViewPicture or ReadSentence

Decide

6

Hidden Markov Models (HMMs)

Hidden Process Models (HPMs)

States (1 time point) Processes (time window)

Latent state sequence Latent process instances

No external input Use experiment design and behavioral data

State transition matrix Process-specific timing distributions

State-specific emission distributions

Process-specific response signatures

1 hidden Markov chain governs observed data

Process instances can overlap in space and time

Forward-backward training algorithm

EM training algorithm

7

ViewPicture in Visual Cortex

8

ReadSentence in Visual Cortex

9

ViewPicture

10

ReadSentence

11

Decide

0 0.5 1 1.5 2 2.5 3 3.5

0 0 0 0 0.025 0.05 0.075 0.85

Seconds following the second stimulus

Multinomial probabilities on these time points

12

Comparing ModelsHPM Avg. Test Set LL

PS -1.0784 * 10^6

PSD -1.0759 * 10^6

PS+S-D -1.0742 * 10^6

PSD+D- -1.0742 * 10^6

PSDB -1.0741 * 10^6

PSDyDn -1.0737 * 10^6

PSDyDnDc** -1.0717 * 10^6

PSDyDnDcB -1.0711 * 10^6

5-fold cross-validation, 1 subject

P = ViewPicture

S = ReadSentence

S+ = ReadAffirmativeSentence

S- = ReadNegatedSentence

D = Decide

D+ = DecideAfterAffirmative

D- = DecideAfterNegated

Dy = DecideYes

Dn = DecideNo

Dc = DecideConfusion

B = Button

** - This HPM can also classify Dy vs. Dn with 92.0% accuracy. GNBC gets 53.9%. (using the window from the second stimulus to the end of the trial)

13

Conclusions

• Simultaneous estimation of spatial-temporal signature (HRF) and temporal onset of cognitive processes.

• Framework for principled comparison of different cognitive models in terms of real data.