MiniTutorial in Computational Neurolinguisticsclic.cimec.unitn.it/brian/compNeuroWSnaacl10/miniTutorialMurphy.pdf · MiniTutorial in Computational ... • Language fluent, ... Wavelet

MiniTutorial in Computational Neurolinguistics

CrashCourse for Computational Linguists in Neuroimaging, Neurolinguistics and Computational Neuroscience

Brian MurphyComputation, Language and Interaction Lab

2

NAACL, LA, June 2010 Brian Murphy, U. of Trento

Overview

• Basics of Brain Anatomy and Function• Neuroimaging techniques and what they detect• Dipping into the neurolinguistics literature• Machine learning with neural data• PostScript

3


Brain Anatomy and Function

4


Brain Architecture

• The brain contains ~100 billion neurons (nerve cells)

• The “primitive” parts of the brain form the connection to the rest of the body (via the brain stem and spinal cord), and control emotions, and involuntary processes (heartbeat, digestion, breathing, balance and coordination of movements etc)

• Most interesting cognition happens in the cerebrum

• http://www.pbs.org/wnet/brain/3d/

Image: howstuffworks.com

http://www.pbs.org/wnet/brain/3d/

5


Brain Architecture

• The cerebrum consists of:– surface grey matter or cortex (computation)– underlying white matter (connecting wiring)

• In humans and larger mammals the surface is folded to maximize the amount of grey matter

• There are two anatomicallysymmetrical hemispheres

• Mostly, these are functionally symmetrical

6


Neurons

• Pyramidal neurons are the nerve cells that are found in the cortex

• Typically have tens of thousands of connections

• In cortex neurons are organised into about 2 million columns of ~5000 cells each (Johansson and Lansner 2007)

• Columns are oriented perpendicular to the surfaceof cortex

Image: neurosciencerus.org

Image: brainmaps.org

7


Neurons• Information is passed by chemical

messengers at the synapses

• Voltage (potential) accumulates in the cell body in response to inputs from excitatory and inhibitory dendrites

• Once a threshold is reached, an action potential is transmitted down the axon, to “fire” the synapses

• Firing rates vary, but often order 50Hz, consuming oxygen and glucose

Images: Lodish et al, Molecular Cell Biology

8


Neuroimaging Techniques

9


Magnetic Resonance Imaging (MRI)

• Reveals static anatomy and dynamic function at high spatial resolution

• Cryogenic superconducting highfield magnet cooled with liquid helium

• Expensive: ~€250/hr

• Loud, bit claustrophobic

10


Magnetic Resonance Imaging (MRI)

• Single protons aligned by standing magnetic field

• Resonant radio waves “kick” protons off spin access

• Radio waves echo back as protons return to alignment

• Rate of realignment reflects characteristics of tissue

• Signals can be tuned tohighlight anatomy or bloodoxygen levels

Images: mridoc.com

e in

e out

11


Functional MRI (fMRI)

• Busy parts of the brain ask for more glucose and oxygen, and the body overcompensates blood supply

• Localised increases in oxygen levels can lead to tiny changes (~1%) in signal

• MRI tuned to oxygen levels = fMRI• BOLD response is sluggish, and

lags by several seconds• Temporal resolution: 1s/5s• Spatial resolution: 15mm• Thousands of voxels per volume

Images: www.fmrib.ox.ac.uk; nri.gachon.ac.kr

http://www.fmrib.ox.ac.uk/

12


fMRI Data

13


fMRI Analysis

• Sensitivity to conditions discovered with correlation

14


Electroencephalography (EEG)

• Coordinated electrical activity from many individual neurons (min ~10k) can lead to largerscale currents in the skull cavity

• These cause voltages at the scalp, strongest if the originating current is radial to the skull

• More direct measure of neural activity, but mixed sources, attenuated by skull

15


Electroencephalography (EEG)

• Typically 64, 128 electrodes record μV changes

• Conductive gel makes connection

• Often in electrically shielded room• Temporal resolution: <10ms

• Hundreds of samples per second• Spatial resolution: 12cm

• Cheap: ~€25/hr

• Preparation time: ~1hr

16


EEG Signals

17


Magnetoencephalography (MEG)• Same principles as EEG, but detects corresponding magnetic fields

• Cryogenic superconducting sensors (SQUIDS)• Strongest for currents tangential to scalp

• Finer spatial sensitivity, so not as deep

• Lack of attenuation = better source individuation

• In magnetically shielded room• Little preparation

time

• Expensive:~€200/hr

Images: Elekta; Christoph Braun

18


Neuroimaging Summary

• When millimetre scale collections of neurons become more active, they ask for more blood – this change becomes visible within several seconds and can be detected with fMRI

• When the activity of centimetre scale collections go into, or fall out of coordination, this electrical signal is visible with EEG within milliseconds

• MEG has someadvantages of both, but still expensive

• EEG/MEG and fMRI/EEG can berecorded simultaneously

Images: Christoph Braun

EEG

19


A sample from the neurolinguistics literature

20


Broca and Wernicke

• Broca's Area (1861): language production; structure

• Wernicke's Area (1875): language comprehension; meaning

• Brain damage is often extensive, and deficits various, and people adapt, making generalisation difficult

Dronkers 2007, Brain

21


Semantic Impairments

• French company director, stroke at 63

• Language fluent, apparently intact other than problems finding words

• Particular problems with food

• Normal performance on synonym matching and picture similarity

• Problems in picture naming, and picture/word matching

Samson & Pillon 2003, Cognitive Neuropsychology

22


Language is Complex

• Single neurons specific to particular people and places

Friederici 2002, Trends in CS

23


Localising Language Centres

Embick et al 2000, PNAS

• Groups of stimuli that should differentially engage modules

• See which areas of the brain “light up” are more active

24


Localising Language Centres

Embick et al 2000, PNAS

25


Localising Lexical SemanticsHotspots

Pulvermüller 2005, Nature Reviews Neuroscience

Overlappingrepresentation

Modal,distributed

26


Detecting Language Violations

• EEG pattern (N400), seen for implausible continuations

• Not just language internal semantics, realworld knowledge too

Kutas & Hillyard, 1980 Hagoort et al 2004, Science

27


Detecting Language Violations

• EEG pattern (P600), seen for syntactic parsing problems

Francesco Vespignani

Ho visto la figlia del macellaio che era partita per Roma.Ho visto la figlia del macellaio che era partito per Roma.Ho visto il figlio del macellaio che era partita per Roma.

• … but can be triggered by semantics too (dependency/ thematic structure)

Kim & Osterhout 2005, J of Memory and Language

28


Scale: A Reality Check

• Single neurons specific to particular people and places (issue of scale)

Quiroga et al, 2005, Nature

29


Machine learning with neural data

30


Data-Mining Neural Sources

Neural Signals

(Networks of)Sources

Image: André Kaup, Uni Erlangen/Nürnberg

ConventionalClassifier ?

31


Data-Mining Neural Sources

Neural Signals

(Networks of)Sources

ConventionalClassifier

32


fMRI: Preprocessing

• fMRI data is noisy in space, time and amplitude• People don't stay still during experiments• Brains vary in size, shape and functional localisation

• SPM or FSL (free packages):– Normalisation of brain dimensions– Spatial and temporal smoothing– Coregistration of functional and anatomical images– Coregistration across participants for group analyses

33


fMRI: Preprocessing

34


fMRI: Preprocessing

35


fMRI: Preprocessing

36


fMRI: Preprocessing

37


fMRI: Extraction and Selection

• Typically can simply take voxel activation samples as feature, as they are assumed to vary monotonically with task

• May use percentsignalchange: normalise signals relative to preceding baseline period, and shift in time to account for BOLD delay

• Or βvalues from GLM: BOLD response is modelled

• Main issue is selection among many thousands of informative, redundant and uninformative voxels

38


M/EEG: Preprocessing

• M/EEG data is noisy in space and amplitude• M/EEG data is affected by brainexternal artefacts• M/EEG data is a mixture of neural activities• EEG data is attenuated and blurred by skull and tissue

• EEGlab (free Matlab package) for:– signal filtering– ICA isolation of artefacts– Laplacian sharpening

39


M/EEG: Preprocessing

• Highpass filter <1Hz removes slow drifts from signal (head/body movements)

• Low pass filter removes electrical and some muscle noise

• ICA used to isolate and remove eyemovements

40


M/EEG: Time Domain

• Reduce granularity of waveform by mean/slope

• Only take significant sequences of samples

41


M/EEG: Freq Domain

Any digitised signal waveform can be equivalently represented by a collection of component frequencies, of particular amplitude and frequency

Wavelet or FFT algorithm

Image: signalworks.com.br

42


Spectral Decomposition

43



44



45


M/EEG: Source Separation

EEG Signals

(Networks of)Synchronous Sources

Image: adapted from Parra et al, 2005, NeuroImage

GenericClassifier

Murphy et al, under review, Brain and Language

46


Machine Learning Packages

• Princeton MVPA:– Matlab package for fMRI

• Dartmouth PyMVPA– Historically related Python package– Custom code for fMRI– M/EEG code under development – Language/text friendly programming

environment– Wide range of machine learning

methods, internal and external

• … or any package you know of that can read Matlab data files

47


Post-Script

48


Advantages for Cognitive Sciences

• Beyond identifying neural correlates of cognitive states, machine learning (=successful prediction) gives a degree of studyinternal replication

• Increased sensitivity = ...– fewer participants, stimuli, trials– single participant analyes > individual differences vs group

commonalities– stimulus level analyses = more nuanced theories

• Neural data has the freedom of bias we get from corpus data, and the speaker understanding we get from elicited data

• EEG may soon be radically cheaper and more portable, allowing experiment in realwolrd contexts

49


Still Plenty To Do

Have any of these phenomena been decoded from neural activity?

– Part of speech– Verb classes– Grammaticality/acceptability– Auditory input, phonetics– Morphological structure– Language production– Continuous comprehension

50


Science Fiction?

Topic is in its infancy... but in X years/decades ...

Neural Parsing?

Brain Dictation?

Word BrainNet?

Image: Emotiv.com

51


ThanksReadings:

– A Comprehensive Model of Language ComprehensionFriederici, 2002: Towards a neural basis of auditory sentence processing, Trends in Cognitive Sciences

– Controversy on Function and Modularity of Broca's RegionGrodzinsky et al, 2008: The battle for Broca's region, Trends in Cognitive SciencesWillems et al, 2009: Broca's region: battles are not won by ignoring half of the facts, Trends in Cognitive Sciences

– Decompositional CorpusLexical Semantics, fMRIMitchell et al, 2008: Predicting human brain activity associated with the meanings of nouns, Science

– Categorial Semantics, EEGMurphy et al, to appear: EEG decoding of semantic category reveals distributed representations for single concepts, Brain and Language

– Software for Machine Learning with Neural RecordingsHanke et al, 2009: PyMVPA: a unifying approach to the analysis of neuroscientific data, Frontiers in Neuroinformatics

– Place of Linguistic Theories in Neuroscience of Language, and viceversaPoeppel et al, 2005: Defining the relation between linguistics and neuroscience in Cutler (ed)Marantz, 2005: Generative linguistics within the cognitive neuroscience of language, Linguistic Review

Documents

MiniTutorial in Computational Neurolinguisticsclic.cimec.unitn.it/brian/compNeuroWSnaacl10/miniTutorialMurphy.pdf · MiniTutorial in Computational ... • Language fluent, ... Wavelet