Workshop on Medical VR and AR€¦ · 0 2 4 6 8 10-1-0.5 0 0.5 1 Time (s) 0 2 4 6 8 10-1-0.5 0 0.5...

Stanford University

April 5, 2018

Workshop on Medical VR and AR

To register go to:

https://scien.stanford.edu/index.php/medicalvrar/

Panel discussions with

Stanford physicians who are using VR and AR applications

Talks by researchers who are

developing VR and AR technologies to advance healthcare

Interactive demos featuring

research projects, clinical applications and startup ventures.

Human neuroimaging using MRI: Methods and stories

Quantitative Measurements

∞Computational Models

∞Check and Share

Brian A. WandellStanford Neurosciences Institute

Stanford Center for Cognitive and Neurobiological ImagingStanford Center for Image Systems Engineering

Presenting the work of many people from my lab at Stanford

Outline

• A very brief introduction to human brain structures

• The MRI instrument: basic signals and image formation

• Interesting case studies using MRI

• MRI diagnostic tools for psychiatry: Reading impairment

Even simple judgments –

such as lightness - depend

on substantial interpretation

of the image data carried out

by brain circuits

Cognitive neuroscience is the

study of how the brain

performs these actions

(Anderson and Winawer,

Nature, 2005)

Cortical computational elements

5

Brain computations depend

on a variety of cells; an

important class of these cells,

the neurons, have their cell

bodies located in the cerebral

cortex (gray matter).

The cortex is a sheet (2-4

mm thick) of tissue that

covers the surface of the

brain; other subcortical

regions and types of cells

matter too!

image from Graham Johnson

Neurons/mm3: 104-106

Cortical Neurons: 1011

Synapses/neuron: 103

Cortical Synapses: 1014

Surface area of each

hemisphere: 20 x 30 cm2

Neuron: impulse-conducting cell; bodies are in the cerebral cortexAxon: a thin fiber that carries the output impulses from a neuronDendrite: a branching process of a neuron that receives impulses from other neuronsSynapse: The point of connection between neurons

Long-range communication architecture (tracts)

Courtesy Professor Ugur Ture

• There are many long-range connections

• These connections are not passive – they change their properties in response to use

• A system with active wires

The human brain

macaque

human

mouse

0.5 g 1500 g100 g

Image adapted from J. Horton

1: 15: 3000 (volume ratios)

• Brains differ

• Check which system was measured

Gross brain anatomy

Brain volume - 1000 – 1500 cm3

Cerebral cortex area - 1200 cm2 (20 x 30 cm) x 2Cerebellum area - 500 cm2

Vision –Occipital and temporal

Hearing –Temporal

Reading –Temporal, parietal, Frontal

The brain is studied at an enormous range of spatial scales

• There is no reliable

theory or model that

relates measurements at

different scales

• This doesn’t stop

scientists from

speculating or speaking

hopefully about

relationships

MRI instrumentation

MRI instrumentation can be

used in many ways to learn

about brain tissue, structure,

and activity

Schematic of MR scanner and magnetic field lines

Liquid helium

Magnetic field lines

http://www.magnet.fsu.edu/education/tutorials/magnetacademy/mri/fullarticle.html

Magnetic resonance signal: The free induction decay experiment

Beaker of water in a perfectly uniform magnetic

field

RF signal excites spins

CoilBloch Purcell

This is the voltage signal we measure and interpret

RF signal is detected

Using a gradient to resolve the spatial locations of two beakers of water

• In a uniform magnetic

field, the free induction

decay produces a single

frequency response from

the two beakers

Magnetic gradients are used to resolve spatial signals

0 2 4 6 8 10-1

-0.5

0

0.5

1

Time (s)

0 2 4 6 8 10-1

-0.5

0

0.5

1

Time (s)

Gradient

• If we introduce a magnetic

field gradient across the two

beakers, the emissions are at

different frequencies;

frequency encodes position

• This principle is developed

into a much more complex

methodology to form images

rapidly as gradients are

switched

The emission frequency varies with local field

MRI instrumentation occupies a couple of rooms

• To measure RF signals in the

living human requires a

substantial set of calibrated

and carefully controlled

components

• The main magnet is super-

cooled (helium) so that the

coils are injected with current

once and no further energy is

needed to sustain the magnet

(Machine room)

Modern MRI scanners human (A-C) and small bore animal (D)

• MRI scanners are

noninvasive

• The RF coils and gradients

are programmable, which

produces many different

types of measurements

• We are still learning ways to

create and interpret MRI data

GE Siemens

Philips

(D)

Bruker

MRI indirectly measures activity

• Blood oxygen level dependent (BOLD) signal

Neural and vascular activity are coupled

Source of

noise

Walter K.

“On turning down a left occipital bone

flap, a large angry-looking angioma

arteriale racemosum of the left occ.

Lobe was disclosed which extensively

involved the visual cortex.

The haemorrhage occasioned by the

bone flap was so excessive that the

operation had to be abandoned

without touching the tumour.

A decompression, however, was made.

The patient was discharged … with

greatly improved vision.”

Observations Upon the Vascularity of the Human

Occipital Lobe During Visual Activity

J.F. Fulton, M.D. (1928)

Neural and vascular activity are coupled

• Subject noted that ‘the noise in the

back of his head increased in intensity

when he was using his eyes.’

• No increase for hearing, touch or

smell

• Increased more when he tried

harder

Source of

noise

Walter K.

Observations Upon the Vascularity of the Human

Occipital Lobe During Visual Activity

J.F. Fulton, M.D. (1928)

Conclusion: Blood flows to the active parts

of the brain when we use a sense (vision).

The blood is an indicator of neural activity.

MRI scanners can measure the blood oxygen level

B0 RF

• There is more oxygenated

blood in regions of the brain

that are active compared to

control regions

• If we alternately present a

stimulus and take it away, the

scanner signal measures the

small, spatially localized,

modulations of the blood oxygen

level over time

Macular stimulation

Perimacular stimulation

Remarkable progress from PET to advanced MRI in 25 years m

m

1986

Voxel size

mm

Fox et al., 1986Nature

(Wandell and Winawer, 2011)

4020

1052.5

1 cm

2.5 5 10 20 40

Eccentricity (deg)

V1

V2

V3

2009

mm

mm

Voxel size

Human eccentricity mapping with fMRI

(Engel et al., 1994,1997; Sereno; Tootell, DeYoe; Others)

• Inflated brain• Gray/white

are sulci/gyri

Pseudo-color representation of visual field map

Angular measurements delineate visual field map boundaries

Combining eccentricity and angle data yields maps

Visual field map reviews366 Neuron 56, October 25, 2007

Vision Research 51 (2011) 718-737

• Maps tile the occipital lobe

• Extend into IPS and VOT

• Response properties differ

• Identification from gross anatomy

Quantitative modeling: the population receptive field (pRF)

‘Responses can be

obtained in a given optic

nerve fiber only upon

illumination of a certain

restricted region of the

retina, termed the

receptive field of the fiber

(Hartline, 1936)’.

+ On0 Off

• Functional description

• Stimulus-referred

Sherrington, 1910Kuffler, 1953

Population receptive field idea

x

y

time

Stimulus

Time (sec)s1

x

y

(x1,y1,s1)

Parameters

% B

OL

D

Predicted BOLD (including HRF)

Observed

• For each voxel, find a

spatial receptive field that

explains the fMRI

measurement.

• The spatial RF model is

the object of interest.

• Minimally, the model is

linear in contrast and has

an (x,y) location in the

visual field and a spread

• More complex models are

also being studied (e.g.,

CSS)

Population receptive field idea

x

y

time

Time (sec)s1

x

y

(x1,y1,s1)

Parameters

% B

OL

D

Predicted BOLD (including HRF)

Observed

Stimulus

• For each voxel, find a

spatial receptive field that

explains the fMRI

measurement.

• The spatial RF model is

the object of interest.

• Minimally, the model is

linear in contrast and has

an (x,y) location in the

visual field and a spread

• More complex models are

also being studied (e.g.,

CSS)

Population receptive field idea

x

y

time

Stimulus

Time (sec)s1

x

y

(x1,y1,s1)

Parameters

% B

OL

D

Predicted BOLD (including HRF)

Observed

Stimulus

• For each voxel, find a

spatial receptive field that

explains the fMRI

measurement.

• The spatial RF model is

the object of interest.

• Minimally, the model is

linear in contrast and has

an (x,y) location in the

visual field and a spread

• More complex models are

also being studied (e.g.,

CSS)

PRF size varies substantially and regularly across visual cortex

pR

F s

ize

(deg

)

pRF eccentricity (deg)

• At common eccentricities, different maps have different pRF sizes• PRF size increases with eccentricity for all maps• Bands are bootstrap estimates of the standard error

• Attention

• Stability and Plasticity

• Prosopagnosia

• Development and aging

• Autism

• Alzheimer’s disease

Trends in Cognitive Sciences, June 2015, Vol. 19, No. 6 349

Image-computable cortical models – Winawer lab

Neurological surprises

The case of a missing optic chiasm (Hoffman et al., 2012)

LH

Right visual field Left visual field

The axonal pathways from the eye to brain

• The optic nerve from each eye meets in the optic chiasm

• Half the fibers cross to the other side of the brain

(seen from bottom)

• The chiasm is visible in a standard anatomical image

The optic chiasm

Diffusion MRI (dMRI)

• We can measure the axonal pathways using another MRI modality, diffusion MRI (dMRI) coupled with computational algorithms (tractography)

• Eye movement (nystagmus)

• Discovered during routine testing for nystagmus

• Resolved after a few weeks

A subject without a chiasm (Achiasmic)

Right visual field Left visual field

Achiasmic axonal pathways from the eye to brain

• For this subject, the optic nerves do not cross!

• The right eye sends signals only to the right brain, and the left eye only to the left brain

(seen from bottom)

Left eye Right eye• Slight decrease in visual

acuity

• Slightly reduced

peripheral visual fields

• No stereopsis

• Prominent infantile and

see-saw nystagmus

(resolved)

Subject AC2 characteristics

FMRI confirms that the right and

left visual fields are overlaid in cortex

Right hemisphere Left hemisphere

L hemisphere

Left visual field

FMRI confirms that the right and

left visual fields are overlaid in cortex

Right hemisphere Left hemisphere

L hemisphere

Right visual field

Modeling the time course (1 Gaussian)

Modeling the time course (2 Gaussians)

Achiasmic - Folded representation

V1

ControlRight retina

Temporal

LH

Right hemisphere

Achiasmic - Folded representation

V1

AchiasmicRight retina

Temporal

LH

Right hemisphere

Summary – the system view

1. A genetic defect that

disrupts crossing at the

chiasm signaling causes a

developmental

reorganization in visual

cortex.

2. Despite the profoundly

disrupted V1 maps, the

rest of the brain figures

out what to do.

Brain plasticity

Specializations of brain function

Wandell et al. 2007, New Encyclopedia of Neurosciences

Reading-

specific loss

Damage to small regions of

gray matter can produce very

specific cognitive problems,

such as face-blindness, loss

of color vision, loss of motion

perception, or loss of

reading ability

Brain plasticity and stability

In development, timing Matters: The search for miracle cures

Recovery from early blindness(Gregory and Wallace, 1963)

Gregory’s patient SB

• Born in 1900, lost site in both eyes

because of corneal infections

• Prior to 2 years of age; kept

bandaged to reduce puss

• Went to a school for the blind to

learn a trade; married

• Received a corneal graft in London

at the age of 52

Gregory’s patient SB

Quite recently he had been

struck by how objects changed

their shape when he walked

round them. He would look at a

lamp post, walk round it, stand

studying it from a different

aspect, and wonder why it looked

different and yet the same.

(Gregory, 1974, p. 111)

Michael May(images courtesy Michael May)

Images from Michael May, Sendero Group

• Chemical explosion (3 yrs old)

• One eye lost; other cornea (and limbic stem cells) destroyed

• Blind (no contrast or form) from age 3 through 46

Recovered sight?(images courtesy Michael May)

Limbic stem cells and corneal replacement

84 m

• Similar to controls at

low spatial frequency

• Substantially worse

above 0.25 cpd

• Constant for the 7

years following surgery

Controls

MM

Perceptual contrast sensitivity functions

Specializations of brain function

Wandell et al. 2007, New Encyclopedia of Neurosciences

Reading-

specific loss

Damage to small regions of

gray matter can produce very

specific cognitive problems,

such as face-blindness, loss

of color vision, loss of motion

perception, or loss of

reading ability

MOTIONMOTION

14 deg

Ca

PO

MM

Ca

PO

Control

MM has an unusual cortical map

MM

AAB

Motion selective cortex

• Responds powerfully

• Is organized as a map

• Has the same size as in

controls

14 deg

LiGFuG

Control

Object and face-related responses

LiG

FuG

Posterior

Medial

Diagnosing the reading circuitry

VWFAV1

Retina

LGN

Optics

PMK

K

Extrastriate

Pulvinar

VWFA

Wernicke

BrocaSTG

IPS

SLF

Arcuate

VOF

ILF

Locating reading circuits and maps

VWFA - essential for reading, but not unique to reading

Measuring the activity while reading (fMRI)

We can see the locations of the cortical activations during reading

Through the maps and on to the VWFA

Field of view in reading circuitry of a single subject

0 1.00.5

5°10°

Sub 20

10020 60

% variance explained by

pRF model (word stimuli)

Using pRF methods,

we have learned that

the portion of cortex

engaged in reading

only sees a small part

of the visual field

This may be why it is

very hard to read in

the peripheral field

Small field of view for the reading circuitry

15 deg

5 deg

Le et al. 2017 Journal of Vision

Left and right hemispheres

Using pRF methods,

we have learned that

the portion of cortex

engaged in reading

only sees a small part

of the visual field

This may be why it is

very hard to read in

the peripheral field

Field of view of the VOT reading reading circuitry

• There are significant differences between participants

• We are correlating these differences with measures of word recognition

• With colleagues we are studying how the FOV in Israeli readers

Sub01 Sub02 Sub03 Sub04 Sub05

Sub06 Sub07 Sub08 Sub09 Sub10

Sub11 Sub12 Sub13 Sub14 Sub15

Sub16 Sub17 Sub18 Sub19 Sub20

5°10°

Left hemisphere

only

Diagnosing the reading circuitry

Long-range neural connections for reading

Inferior fronto-occipital fasciculum

150 Directions, 2 mm3, B=2000 projected on a 1 mm3 T1 anatomical image

Franco Pestilli 2014 - Stanford University

White matter fascicles are generatedby step-wise sampling of local diffusion information

Left IFOF

VWFAV1

Retina

LGN

Optics

PMK

K

Extrastriate

Pulvinar

VWFA

WernickeBroca

STG

IPSSLF

ArcuateVOF

ILF

Major components of the reading pathway

Wandell and Le (2017)

Learning to See WordsB.A. Wandell, A. Rauschecker and J. Yeatman (2012).Annual Review of Psychology Vol. 63, pp.31-53.

The goal: Diagnosis

Identifying the locations

and responses in a poor

reader that differ

significantly from

measurements in good

readers

Diffusion (FA) changes differs between good and poor readers

• Measured brain and behavior at 4 time points (data management!)

• The first measurements predict reading over the next few years

• The rate and direction of FA development differs between good and poor readers in both the Arcuate and the ILF

Blue: Good readersRed: Poor readers

More linear

(FA)

More circular

Fractional anisotropy (displaced)

Diffusion (FA) changes differs between good and poor readers

Mean FA

development

slopes

Left ILF

Fractional anisotropy

Age• Measured brain and behavior at 4 time points (data management!)

• The first measurements predict reading over the next few years

• The rate and direction of FA development differs between good and poor readers in both the Arcuate and the ILF

Correlations between tract diffusion change and seeing words

• Development measured

by dMRI in the ILF and

Arcuate, but not others

tracts, correlates with

the ability to rapidly see

words

• This is one reason we

think that the wires are

active, changing in

response to learning and

memory

r = 0.51

(Yeatman et al., 2012, PNAS)

Mea

sure

d r

ead

ing

sco

re

Measured FA development rate

Neuroprognosis

• Simple models that combine tissue

properties from two tracts (ILF and

AF) predict measured reading skill

• The predictions are not yet useful;

they are statistically reliable

r = 0.66 (43%)7-11 yrs

Predicted reading score

Mea

sure

d r

ead

ing

sco

re

Predicting reading scores from rate of white matter development

(Yeatman et al., 2012)

Connectionism: Mismatch hypothesis

VOT Specialized processing for faces, words, other things

General visual inputs

Connectionism: Mismatch hypothesis

General visual inputs

Computational neuroimaging: Reading circuitry

• We have made progress in computational

neuroimaging methods, so that we have the maps

and some computational methods for key

properties (pRF)

• We can follow responses to words up to VOT cortex

in living human subjects at mm resolution; using

dMRI we can identify the major tracts that carry

these signals and that the cortex learns to recognize

words using these circuits

• We hope to build computational models based on

these MR measurements of the reading circuits,

relating the neuroimaging measures to behavior,

and to understand the biological reasons for

success and failure of the reading circuitry in each

child

Summary

• A very brief introduction to human brain

structures

• The MRI instrument: basic signals and

image formation

• Interesting case studies using MRI

• MRI diagnostic tools for

psychiatry: Reading impairment

Thanks to NIH, NSF, Simons, Weston-Havens, Wallenberg Foundation

HiromasaTakemura

Franco Pestilli

Jason Yeatman

Ariel Rokem

Kendrick Kay

Jon Winawer

Alyssa Brewer

Michal Ben-Shachar

Serge Dumoulin

Rosemary Le

Andreas Rauschecker

BobDougherty

Gunnar Schaefer

Michael Perry

YoichiroMasuda

Hiroshi Horiguchi

Heidi Baseler

Alex Wade

Anthony Morland

Stephen Engel

Stanford University

April 5, 2018

Workshop on Medical VR and AR

To register go to:

https://scien.stanford.edu/index.php/medicalvrar/

Panel discussions with

Stanford physicians who are using VR and AR applications

Talks by researchers who are

developing VR and AR technologies to advance healthcare

Interactive demos featuring

research projects, clinical applications and startup ventures.