Carnegie Mellon 1. Toward Exploiting EEG Input in a Reading Tutor Jack Mostow, Kai-min Chang, and Jessica Nelson Project LISTEN listen

Carnegie Mellon

1

Carnegie Mellon

2

Carnegie Mellon Toward Exploiting EEG Input in

a Reading Tutor

Jack Mostow, Kai-min Chang, and Jessica Nelson Project LISTEN

www.cs.cmu.edu/~listenCarnegie Mellon University

This work was supported by the Institute of Education Sciences, U.S. Department of Education, through Grants R305A080157 and R305A080628 to Carnegie Mellon University. The opinions expressed are those of the authors and do not necessarily represent the views of the Institute or U.S. Department of Education.

Carnegie Mellon

Motivation: peer into student’s mind• Identify mental states

– Cognitive: effort, recognition, understanding, learning, …– Affective: attention, engagement, frustration, …– Motor: speech, motion, expression, …

• Decide accordingly– What to teach next– How to teach it– …

3

Carnegie Mellon

Peering Into Minds: EEG (electroencephalography)

4

Carnegie Mellon EEG in the Lab

5

Cons:• Impractical for schools!• Wet electrodes (gel or saline)• Expert application and monitoring• Money for equipment & personnel• Magnetically-shielded room

Pros:• Controlled experimental conditions• Temporally fine-grained data• Reflects widespread brain activity• Detects relevant mental states: attention

(Marosi et al. ’02), engagement (Lutsyuk et al. ’06), frustration (Berka et al. ’06)

Carnegie Mellon

New Portable EEG Devices

6

Pros:• Feasible to use in schools• Inexpensive• Dry electrodes• No expert needed• Headphones and microphone

Cons:• 1-2 electrodes; not whole brain• What can such devices detect?

Carnegie Mellon

Detect brain states related to learning?

7

1. Can EEG detect when reading is difficult?2. Can EEG detect differences between words?3. Which EEG features detect differences best?

Carnegie Mellon

Easy vs Hard (Grade K-1) (GRE level)

8

We need water, land, and air to live. Earth has all these things. Water covers much of Earth. Most of this water is not safe to drink. Many people are running out of fresh drinking water.

In regard to propaganda the early advocates of universal literacy and a free press envisaged only two possibilities: the propaganda might be true, or it might be false. They did not foresee what in fact has happened…

Carnegie Mellon 1. Detect when reading is difficult?

Carnegie Mellon 1. Detect when reading is difficult?

• For adults / children– 10 adults in Project LISTEN lab– 11 children ages 9-10, at school

• Reading connected text / isolated words• Aloud / silently

9

Carnegie Mellon

Child readerAdult reader

Pilot study setup

10

Carnegie Mellon

Project LISTEN’s Reading Tutor

Reader

Carnegie Mellon Collecting EEG and Reading Tutor Data

11

Time Student Text07-12-2011-09:51:00.10 Kai-min We …07-12-2011-09:51:00.20 Kai-min need …07-12-2011-09:51:00.35 Kai-min need …07-12-2011-09:51:00.55 Kai-min water…07-12-2011-09:51:01.01 Kai-min land …07-12-2011-09:51:01.50 Kai-min air …

Reading Tutor LogTime Student Raw Attention07-12-2011-09:51:00.01 Kai-min -33 11 …07-12-2011-09:51:00.02 Kai-min 351 17 …07-12-2011-09:51:00.03 Kai-min 117 20…07-12-2011-09:51:00.04 Kai-min 66 1 14 …07-12-2011-09:51:00.05 Kai-min -451 26…07-12-2011-09:51:00.06 Kai-min -3 43…

MindSet (EEG) LogTime Student Text Raw07-12-2011:09:51:00.10 Kai-min We -33 …07-12-2011:09:51:00.20 Kai-min need 351 …07-12-2011:09:51:00.35 Kai-min need 117 …07-12-2011:09:51:00.55 Kai-min water 661 …07-12-2011:09:51:00.10 Kai-min land -451 …07-12-2011-09:51:01.50 Kai-min air 43 ,,,

Combined Log

Carnegie Mellon MindSet (EEG) Features

12

Raw EEG signal, reported at 512 Hz

Filtered EEG signal, 512 Hz

Proprietary “attention” measure, 1 Hz

Proprietary “meditation” measure, 1 Hz

Proprietary signal quality measure, 1Hz

Theta band (4-7 Hz), 8Hz

Alpha band (8-11 Hz), 8Hz

Beta band (12-29 Hz), 8Hz

Gamma band (30-100 Hz), 8Hz

Gamma+ band (101-256 Hz), 8Hz

Delta band (1-3 Hz), 8Hz

Carnegie Mellon

13

Machine Learning Approach

• Train classifiers to detect mental states associated with stimuli– f = Binary Logistic Regression Classifier– X = MindSet (EEG) Features (averaged over stimulus interval)– Y = Easy or hard sentences

N

11 F

Y = f( X )N

1

Carnegie Mellon

14

Train on: Test on:

Train on: Test on:

Trial 1Trial 2Trial 3Trial 4




Reader-Specific Classifiers

Reader-Independent Classifiers

Carnegie Mellon Class Size Imbalance

15

• More easy sentences than hard ones

• Made the two sets equal in size using 3 approaches:1. Random Oversampling (with replacement)


16


• Made the two sets equal in size using 3 approaches:1. Random Oversampling (with replacement)2. Random Undersampling


17


• Made the two sets equal in size using 3 approaches:1. Random Oversampling (with replacement)2. Random Undersampling3. Directed Undersampling (Truncating)

Carnegie Mellon

Detect when reading is difficult:Reader-specific classifier accuracy

18

oral

sile

nt

both

oral

sile

nt

both

oral

sile

nt

both

adul

t child both

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

over-sample under-sample truncate

chance

p < .05

Carnegie Mellon

oral

sile

nt

both

oral

sile

nt

both

oral

sile

nt

both

adult child both

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7


Detect when reading is difficult:Reader-independent classifier accuracy

19

chance

p < .05

Carnegie Mellon

2. Detect differences between words?

20

Easy Words

BedroomChickenStation

Hard Words

CologneChassisBrocade

Non-Words

KOFCUNWAF

Illegal String

FFSGHTNKL

Carnegie Mellon

21

Machine Learning Classifiers• Multinomial logistic regression classifiers• Measure classifier’s rank accuracy (Mitchell et al. ’04)

– Use classifier to rank-order possible class labels– Rank accuracy = percentile rank of correct label; 0.5 = chance– More sensitive than % correct

rank accuracy = 0.67

Classifier’s Ranking

1. Non-Word2. Illegal String3. Hard Word4. Easy Word

1.0

0.5

0.0

True label

Carnegie Mellon

Detect differences between words: Reader-specific rank accuracy

22

oral

sile

nt

both

oral

sile

nt

both

oral

sile

nt

both

adul

t child both

00.10.20.30.40.50.60.7


chance

p < .05

Carnegie Mellon

oral

sile

nt

both

oral

sile

nt

both

oral

sile

nt

both

adul

t child both

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7


Detect differences between words: Reader-independent rank accuracy

23

chance

p < .05

Carnegie Mellon

3. Which features detect text difficulty best?

• Train classifier using each feature in isolation• Average accuracy across subjects, CV folds

– Higher = better

DELTATHETA

MED

RAWW

AVE

GAMMA+

FILTERED

ALPHA

BETA

GAMMAATT

0.400.420.440.460.480.500.520.540.56

Reader-specific accuracy

OversampleUndersampleTruncate

Feature

0.400.420.440.460.480.500.520.540.560.58 Reader-independent accuracy


Feature

Carnegie Mellon

3. Which features detect word differences best?

• Train classifier using each feature in isolation• Average rank accuracy across subjects, CV folds

– Higher = better

BETAALP

HAMED

ATT

GAMMA

FILTERED

RAWW

AVE

THETA

GAMMA+DELT

A0.420.440.460.480.500.520.540.56

Reader-specific rank accuracy of feature


Feature

THETA

FILTERED

ATTMED

GAMMA+

GAMMABETA

RAWW

AVE

ALPHA

DELTA

0.42

0.46

0.50

0.54

0.58

Reader-independent rank accuracy of feature


Feature

Carnegie Mellon Which EEG features are sensitive to

which lexical properties?

26

Delta(1-3 Hz)

Theta(4-7 Hz)

Alpha(8-11 Hz)

Beta(12-29 Hz)

Gamma(30-100 Hz)

Gamma+(101-256 Hz)

ConcretenessImageabilityColorado Meaningfulness -0.08 -0.10Familiarity -0.08Age of acquisition Brown verbal frequency -0.12 -0.13 -0.09 -0.12Kucera and Francis written frequency 0.07 0.10Thorndike-Lorge frequency 0.09Number of letters 0.11 0.13 0.10 0.10 0.14 0.16

• Do EEG spectra reflect natural lexical variance among sentences?– If so, those bands may carry different information for a tutor.

• Correlate with MRC Psycholinguistic word properties (Coltheart 81)– Blank = not statistically significant; bold = passes False Discovery Rate test

Carnegie Mellon

4. Conclusions and Future Work

• 1-electrode EEG tells easy from hard better than chance.• Frequency bands tap different properties a tutor may use.• Detect mental states such as attention or frustration• Use longitudinal EEG in schools to:

– Instrument authentic behavior– Label data based on normal tutor use, not artificial experiments – Detect longer-term learning, not just recency effects– Combat EEG noise with “big data”– Inform tutor redesign and make student-specific models

27

Carnegie Mellon

28

Questions?

Carnegie Mellon

29

Carnegie Mellon

3. Which features detect word differences best?

• Train classifier using each feature in isolation• Average rank accuracy across subjects, CV folds

– Higher = better

BETAALP

HAMED

ATT

GAMMA

FILTERED

RAWW

AVE

THETA

GAMMA+DELT

A0.420.440.460.480.500.520.540.56

Reader-specific rank accuracy of feature


Feature

Carnegie Mellon

Which features predict word differences best?

THETA

FILTERED

ATTM

ED

GAMM

A+

GAMM

ABETA

RAWW

AVE

ALPHA

DELTA0.42

0.46

0.50

0.54

0.58

Reader-independent rank accuracy of feature


Feature

Documents

Carnegie Mellon 1. Toward Exploiting EEG Input in a Reading Tutor Jack Mostow, Kai-min Chang, and Jessica Nelson Project LISTEN listen