Detection and online classification of the near-infrared spectroscopy fast

optical signal for brain-computer interfaces

by

Nicole Proulx

A thesis submitted in conformity with the requirementsfor the degree of Master of Applied Science

Graduate Department of Biomaterials and Biomedical EngineeringUniversity of Toronto

c©Copyright by Nicole Proulx 2016

Detection and online classification of the near-infrared spectroscopy fast

optical signal for brain-computer interfaces

Nicole Proulx

Master of Applied Science

Graduate Department of Biomaterials and Biomedical EngineeringUniversity of Toronto

2016

Abstract

Near-infrared spectroscopy (NIRS) can detect a fast optical signal (FOS), corresponding to optical

property changes in neuronal tissue during neuronal activation. The FOS has high temporal and

spatial resolution, but has a low signal-to-noise ratio. The FOS has yet to be automatically classi-

fied, hence its value as a BCI control signal remains unknown. During offline and online sessions,

participants performed a visual oddball task. In offline sessions, the FOS was validated with elec-

troencephalography (EEG) measurements of event-related potential (ERPs). Spectral relationships

between FOS and ERP oddball responses were found in upper delta and theta bands using coherence

and Granger causality metrics. A temporal FOS-ERP correlation was also found 200 ms after oddball

presentation. Offline and online FOS classification of oddballs versus frequent responses was achieved

with average balanced accuracies of 62 ± 5% and 63 ± 6%, respectively. FOS classification results

were above chance but did not reach the threshold (>70%) for effective BCI communication.

ii

Acknowledgments

I would first like to thank my thesis supervisor, Dr. Tom Chau, for his guidance and support

throughout the entire process of completing this thesis. Dr. Chau’s motivation was instrumental in

helping me successfully complete this thesis.

I am thankful for the constructive feedback provided by my committee members, Dr. Azadeh Kushki,

Dr. Ryan Hung and Dr. Anne-Marie Guerguerian. Their thoughtful input has significantly con-

tributed to improving the quality of this thesis.

I would like to thank the members of the PRISM lab for their support and assistance, particularly

Larissa Schudlo for answering my numerous questions and always being willing to help. In addition,

thank you to Ka Lun Tam and Pierre Duez for their technical assistance.

I would like to thank my parents for their continuous support and for always being there for me.

Lastly, I would like to thank donors of the Kimel Family Family Graduate Student Scholarship in

Paediatric Rehabilitation, the Bloorview Research Institute, the National Sciences and Engineering

Research Council (NSERC), donors of the Frank Howard Bursary and the Toronto Rehabilitation

Institute for their financial support.

iii

Table of Contents

1 Introduction 1

1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.2 Research Question and Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.3 Thesis Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2 Background 4

2.1 Brain-Computer Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2.2 Near-infrared Spectroscopy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.2.1 NIRS Instruments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.2.2 Slow Hemodynamic Response . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.2.3 Fast Neuronal Response . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.2.4 NIRS Brain-Computer Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.3 The Fast Optical Signal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2.4 Automatic Classification of Neuronal Activity . . . . . . . . . . . . . . . . . . . . . . 9

2.5 Visual Oddball Task and the Prefrontal Cortex Response . . . . . . . . . . . . . . . . 10

3 Quantifying Fast Optical Signal and Event-Related Potential Relationships Dur-

ing a Visual Oddball Task 11

3.1 Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

3.2 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

3.3 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

3.3.1 Participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

3.3.2 Instrumentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

3.3.3 Task Paradigm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

iv

3.3.4 Experimental Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

3.3.5 Data Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

3.3.5.1 FOS Preprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

3.3.5.2 ERP Preprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

3.3.5.3 FOS-ERP Temporal Landmark Correlations . . . . . . . . . . . . . . 20

3.3.5.4 FOS-ERP Spectral Relationships . . . . . . . . . . . . . . . . . . . . 21

3.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

3.4.1 ERP Response . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

3.4.2 FOS Response . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

3.4.3 FOS-ERP Temporal Landmark Correlations . . . . . . . . . . . . . . . . . . . 24

3.4.4 FOS-ERP Spectral Relationships . . . . . . . . . . . . . . . . . . . . . . . . . 25

3.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

3.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

4 Online Classification of the Near-Infrared Spectroscopy Fast Optical Signal for

Brain-Computer Interfaces 30

4.1 Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

4.2 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

4.3 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

4.3.1 Participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

4.3.2 Instrumentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

4.3.3 Experimental Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

4.3.3.1 Offline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

4.3.3.2 Online . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

4.3.4 Offline FOS Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

4.3.4.1 Preprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

4.3.4.2 Feature Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

4.3.4.3 Feature Selection and Classification . . . . . . . . . . . . . . . . . . . 40

4.3.5 Offline ERP Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

v

4.3.6 Online Feature Selection and Classification . . . . . . . . . . . . . . . . . . . . 43

4.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

4.4.1 ERP results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

4.4.2 FOS Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

4.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

4.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

5 Conclusion 51

5.1 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

5.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

5.2.1 Further investigation of FOS spectral features . . . . . . . . . . . . . . . . . . 52

5.2.2 Measurement of additional cortical locations . . . . . . . . . . . . . . . . . . . 52

5.2.3 Potential FOS-BCI paradigms . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

5.2.4 FOS studies with the target population . . . . . . . . . . . . . . . . . . . . . . 53

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

vi

List of Tables

4.1 Feature types extracted from all individual channels and channel combinations of DC

intensity and phase delay at 690 nm and 830 nm (P2P = peak to peak, + = positive,

− = negative, LLE = largest Lyapunov exponent, PSD = power spectral density). . . 40

4.2 Feature types selected from DC 690 nm and 830 nm feature sets for online classification

during the first online session (session 4). Superscripts denote the number of channels

from which a feature type was selected. The number of zero crossings (feature 8)

and the variance of the 0-500 ms window (feature 30), both shown in bold font, were

selected for nearly all participants in both DC 690 nm and DC 830 nm features sets.

Participant 3 did not attend the online sessions. . . . . . . . . . . . . . . . . . . . . . 46

vii

List of Figures

2.1 NIR photon path between source and detector [10] . . . . . . . . . . . . . . . . . . . . 5

2.2 Absorption coefficients in biological tissue as a function of wavelength [15] . . . . . . 6

3.1 NIRS source-detector configuration. Each empty circle, labeled 1-5, represents 2 NIRS

sources, one at a wavelength of 690 nm and the other at 830 nm. Each filled circle,

labeled A-D, represents a detector. Measurement locations, i.e. channels, are located

half-way between sources and detectors and denoted by a red ’X’. Detector D is on

the participant’s left side. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

3.2 Location of EEG electrode according to the International 10-10 system. EEG elec-

trodes were placed at AF3, AF4, F7, F8, F9 and F10 electrode locations shown in

green. The ground electrode, shown in purple, was placed at AFz and the reference

electrodes, shown in orange, were placed at the left and right mastoids. . . . . . . . . 16

3.3 Experimental protocol for one session. The oddball image of Einstein’s face was re-

placed by a smiley face in this figure for copyright purposes. . . . . . . . . . . . . . . 18

3.4 FOS preprocessing steps for DC and phase delay measurements. . . . . . . . . . . . . 19

3.5 ERP 100-trial average response for participant 7. The red line denotes the oddball

image response and the blue line denotes the frequent image response. Time 0 denotes

the instance of image presentation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

3.6 FOS 400-trial averaged response for channel B3 of participant 14. The red lines denote

the oddball image response, the blue lines the frequent image response. The asterisk

denotes a FOS peak that exceeded the FOS peak threshold, meaning that the FOS

peak was significantly correlated temporally with a corresponding ERP peak. . . . . . 23

viii

3.7 DC 830 nm FOS-ERP correlation in session 1 of participant 6. The top plot shows

the FOS DC intensity oddball response for channel B5 and the bottom graph the

ERP oddball response for channel AF3. The shaded vertical bar denotes the latency

between the significantly correlated FOS and ERP peaks (p < 0.05). . . . . . . . . . . 24

3.8 DC 830 nm FOS-ERP coherence for session 2 of participant 4. (a) FOS-ERP coherence

values. (b) FOS-ERP channel pairs with statistically significant coherence (p < 0.01)

are shown in red. Significant coherence is shown for FOS channels A2, A3, B5 and D4. 25

3.9 ERP to FOS (DC 830 nm) Granger causality for session 2 of participant 5. (a) ERP to

FOS Granger causality values. (b) FOS-ERP channel pairs with statistically significant

Granger causal influences of ERP on FOS are shown in red. Significant Granger

causality is shown between all ERP channels and FOS channels A2, B5 and D5. . . . 26

4.1 NIRS and EEG channel configuration. White circles denote paired 690 nm and 830 nm

wavelength NIR sources, black circles denote detectors and red X’s denote measure-

ment channels. Grey diamonds denote EEG measurement electrode locations and the

red diamond denotes the ground electrode location. . . . . . . . . . . . . . . . . . . . 35

4.2 Data collection session protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

4.3 Online feedback presented after each block during the online runs for (a) successfully

and (b) unsuccessfully classified blocks. . . . . . . . . . . . . . . . . . . . . . . . . . . 37

4.4 Individual classifier results per participant for each FOS feature set. (a) DC intensity

at 690 nm wavelength. (b) DC intensity at 830 nm wavelength. (c) Phase delay at

690 nm wavelength. (d) Phase delay at 830 nm wavelength. An asterisk (*) denotes

a participant with DC intensity classification results significantly higher than phase

delay results. A λ denotes a participant with 830 nm classification results significantly

higher than 690 nm results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

4.5 Online FOS balanced classification accuracy per session. The horizontal line denotes

the chance level for the online sessions (binomial test, p < 0.05). Participant 3 did not

attend the online sessions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

ix

List of Abbreviations

BCI Brain-Computer Interface

EEG Electroencephalography

ERP Event-Related Potential

ERO Event-Related Oscillation

NIRS Near-Infrared Spectroscopy

FOS Fast Optical Signal

EROS Event-Related Optical Signal

SNR Signal-to-Noise Ratio

MFG Middle Frontal Gyrus

IFG Inferior Frontal Gyrus

ICA Independent Component Analysis

FFT Fast Fourier Transform

LDA Linear Discriminant Analysis

SVM Support Vector Machine

FCBF Fast Correlation Based Filter

RBF Radial Basis Function

ROC Receiver Operating Characteristic

x

Chapter 1

Introduction

1.1 Motivation

Individuals with severe motor impairment but capable cognitive function often have no means of

communication [1]. There are several causes of severe motor impairment including amyotrophic

lateral sclerosis (ALS) [1], brainstem stroke [2] and severe cerebral palsy [3]. For children with severe

cerebral palsy, the inability to communicate can greatly affect their daily interactions with family

and peers, and can lead to psychosocial impairment [3]. Similar issues arise in adults with severe

motor impairment as the inability to communicate can contribute to their isolation and dependence

on a caregiver [2]. In recent years, many augmentative and assistive communication (AAC) devices

have been developed, such as buttons, switches and eye-tracking devices. However, many of these

AAC devices are not adequate for individuals with severe to complete motor impairment as they

require some form of repeatable, voluntary movement [4].

Brain-computer interfaces (BCIs) are an access technology allowing individuals to communicate via

decoding of their cognitive activity [5]. BCIs can provide a method of communication for individuals

who are unable to communicate their basic needs due to severe motor impairment [4]. Hence, BCIs

could provide an enhanced quality of life for these individuals. BCIs are mainly evaluated according

to their speed, accuracy and ease of use [6]. To enable effective BCI communication, there is a need

to detect localized cognitive activity rapidly and accurately. However, current BCI modalities, such

1

as electroencephalography (EEG) and near-infrared spectroscopy (NIRS), are unable to provide both

high temporal and spatial resolution. NIRS BCIs currently measure a localized but slow hemody-

namic response but NIRS can also detect a fast optical response, commonly named the fast optical

signal (FOS), which corresponds to changes in the optical properties of neuronal tissue during neu-

ronal activation [7]. The FOS has a temporal resolution on the order of milliseconds in addition to

a spatial resolution of less than 1 cm [8]. The combined high spatial and temporal resolution of the

FOS provides an opportunity for a new BCI modality. Yet, it remains to be determined if the FOS

can be detected both accurately and rapidly in an online classification setting.

1.2 Research Question and Objectives

This research aimed to answer the following questions:

Can the fast optical signal be detected in the prefrontal cortex of the brain during a visual oddball task

using frequency-domain NIRS? If so, at what level of accuracy can online classification of the fast

optical response to oddball images versus frequent images be achieved?

To answer these questions, the objectives of this thesis were three-fold:

1. Validation of the prefrontal FOS response during a visual oddball task through quantifying

temporal and spectral relationships with co-located ERP measurements.

2. Evaluation of the window of observation and multiplicity of trials required for FOS automatic

classification during a visual oddball task.

3. Development and assessment of an online classification algorithm for the FOS response during

a visual oddball task.

2

1.3 Thesis Outline

Following this introduction, chapter 2 provides background information on brain-computer interfaces,

near-infrared spectroscopy, the fast optical signal, automatic classification of neuronal responses and

known prefrontal cortex responses to visual oddball tasks. An offline and an online study were

completed for this research work. The purpose of the offline study was to validate the FOS with

ERP correlations and to develop an offline classification algorithm. Correlation results of the offline

study are presented in Chapter 3, which focuses on the temporal and spectral relationships of ERP

and FOS prefrontal responses during a visual oddball task. The online study evaluated the FOS

classification algorithm with an online BCI paradigm. Classification results from the offline and

online studies are presented in Chapter 4, which focuses on automatic classification of FOS responses

during a visual oddball task. Chapter 5 summarizes the contributions of this thesis and provides

directions for future work.

3

Chapter 2

Background

2.1 Brain-Computer Interfaces

BCIs allow communication or control of an external device by decoding cognitive activity [5]. BCIs

can therefore provide an alternative method of communication for individuals with severe motor

impairment as no movement or speech is required. The most common non-invasive BCI modality

is EEG, which measures the electrical potential of neuronal responses and thus has high temporal

resolution, on the order of milliseconds [4]. Signals measured with EEG include slow cortical po-

tentials, sensorimotor rhythms and event-related potentials (ERPs), such as the P300 [4]. EEG has

been extensively studied due to its high temporal resolution, portability and relatively low cost [5].

However, EEG only offers modest spatial resolution, rendering it difficult to localize the neuronal

response [4]. Another disadvantage of EEG is the use of gel electrodes, which increase setup time

and subject discomfort [9]. fMRI is another BCI modality, which provides higher spatial resolution

by measuring the paramagnetic charge of cerebral blood oxygen level dependent (BOLD) flow [1].

However, fMRI is expensive and non-portable, which impedes its applicability for BCI communica-

tion [1]. NIRS is an optical BCI modality which has become popular in recent years due to its ease

of use, portability, relatively low cost and comparable spatial resolution to that of EEG, on the order

of a centimeter [5]. NIRS instruments and responses will be explained in the following section.

4

2.2 Near-infrared Spectroscopy

2.2.1 NIRS Instruments

Near-infrared light can penetrate to a sufficient depth in the human head to detect changes in cere-

bral tissue. Sources emit NIR light into the head and photodetectors placed a few centimeters from

the sources measure the NIR photon absorption and scattering through layers of extra-cerebral and

cerebral tissue. As shown in Figure 2.1, the average photon path forms a banana shape between

each source-detector pair. The depth of the photon path is equivalent to approximately half the

source-detector distance, the optimal source-detector separation for cerebral measurements is there-

fore around 3 cm as larger separations have a lower signal-to-noise ratio [10]. Each source-detector

pair is referred to as a channel.

Figure 2.1: NIR photon path between source and detector [10]

There are three types of NIRS instruments. Frequency-domain NIRS employs a source modulated

at a frequency on the order of 100 MHz. This allows AC and DC intensity as well as phase shift

measurements. Phase shift or phase delay measurements allow calculation of the photon time of flight

from source to detector [11]. Continuous-wave NIRS uses a constant intensity source, therefore

CW NIRS can only measure changes in intensity. Time-domain NIRS employs very short laser

pulses, on the order of picoseconds, to measure the temporal response of tissue. TD NIRS enables

separate estimation of absorption and scattering, however, such systems are more expensive and

provide a lower signal-to-noise ratio [11].

5

2.2.2 Slow Hemodynamic Response

The slow hemodynamic response, related to blood oxygenation levels, is most commonly measured

with NIRS. The latency of the hemodynamic response is on the order of 4-8 seconds [12]. Through

neurovascular coupling, neuronal activation is known to increase cerebral blood flow and oxygen

absorption in cerebral tissues in the activated region of the brain, which in turn affects oxygenated

hemoglobin (oxy-Hb) and deoxygenated hemoglobin (deoxy-Hb) chromophore concentrations [13].

In the range of wavelengths used by NIRS, between 650 and 950 nanometers, absorption of light in

biological tissue is mainly due to oxy-Hb and deoxy-Hb chromophore concentrations [14], as shown

in Figure 2.2.

Figure 2.2: Absorption coefficients in biological tissue as a function of wavelength [15]

The modified Beer-Lambert’s law, as defined in Equation 2.1, calculates the attenuation and the

absorption coefficient at a specific wavelength

A = log

(IoI(t)

)= αCXd+K (2.1)

where A is the attenuation, Io is the initial light intensity entering the tissue, I is the light intensity

exiting the tissue, α is the absorption coefficient, C is the concentration of the absorber, X is the

differential path factor and K is a geometric term accounting for scattering, which is assumed to

be constant. Changes in oxy-Hb (HbO2) and deoxy-Hb (HB) concentration are calculated using

6

absorption coefficients at two wavelengths, typically 690 nm and 830 nm, using Equation 2.2.

∆C = α−1δA =

α690Hb α690HbO2

α830Hb α830HbO2

−1 ln(I690(t1)/I690(t2))

X690d

ln(I830(t1)/I830(t2))X830d

=

∆CHb

∆CHbO2

(2.2)

2.2.3 Fast Neuronal Response

NIRS can also measure a fast neuronal response, commonly called the FOS, related to changes in

the scattering properties of neuronal tissue during neuronal activation. As it directily related to

neuronal activity, the FOS has a latency on the order of 100 milliseconds [7]. Optical changes were

first detected in isolated neurons and nerves during depolarization by Cohen in 1973 [16]. More

recently, a transient optical response was detected in brain tissue during neuron activation and was

determined to be related to transient cellular volume changes [17]. During neuronal activation, an

increase in membrane potential changes intracellular ion concentrations, which causes water to enter

the neuron and consequently to increase its volume. Scattering of near-infrared light was found to

decrease as neuron cellular volume increased, both at single cell and bulk tissue levels [17]. A decrease

in scattering during neuronal activation has been observed on human subject with NIRS via phase

delay and intensity variations [7].

2.2.4 NIRS Brain-Computer Interfaces

As previously mentioned, NIRS has gained popularity as a BCI modality due to its easy of use,

relatively low cost and portability. The prefrontal cortex has been widely used in NIRS BCI studies

due to the absence of hair in this region, thus reducing noise in the signal. Various mental tasks

have been explored in NIRS BCI literature. Tasks measured in the prefrontal cortex include music

imagery, mental arithmetic, word generation, object rotation and emotion induction [10]. The motor

cortex has also been studied for NIRS BCIs using a motor imagery task [10]. All NIRS BCI studies

found in literature measure the hemodynamic response. The information transfer rate of current

NIRS BCIs is therefore limited due to the latency of the hemodynamic response [12].

7

2.3 The Fast Optical Signal

The FOS has been studied with human subjects for over 20 years. Various names are found in

literature for the FOS including the fast optical response (FOR) [18], the optical neuronal signal

(ONS) [19] and the event-related optical signal (EROS), a common term for the event-locked fast

optical signal. First detected in 1995 with motor and visual stimulations by Gratton et al. [20, 21],

the FOS has since then been studied by a few different groups around the world and has been

detected in various parts of the brain. Cortical areas where the FOS has been studied and detected

include the visual [19, 21–28], somatosensory [18], motor [20, 29–32], auditory [11] and prefrontal

cortices [8,33–40]. The visual and prefrontal cortices are the most studied regions for the FOS.

FOS activity has been detected in the prefrontal cortex using a variety of activation tasks and

stimuli. During visual and auditory oddball target detection, inhibitory processes (No-Go) were

found to increase FOS activity near the middle frontal gyrus (MFG) and inferior frontal gyrus (IFG)

border, whereas counting the oddball targets increased FOS activity in the MFG [35]. During a

Go-NoGo paradigm, target detection was found to increase FOS activity in the right MFG and left

IFG [37]. Task switching between verbal and spatial tasks was found to increase FOS activity in

the left IFG and MFG [8]. Language processing and sentence comprehension were also found to

activate the FOS response in the left IFG [34]. FOS response latencies in the prefrontal cortex have

ranged between 100-350 ms [8, 33–37, 39]. The FOS amplitude in the prefrontal cortex has ranged

between 0.01-0.05% for relative changes in intensity (∆I/Io) [36] and between 0.03-0.2◦ for phase

delay measurements [35,39].

The reliability of the FOS has been disputed due to its low signal-to-noise ratio (SNR). In addition,

the high localization of the FOS increases the difficulty in obtaining a reliable response [29]. At

least 2 studies have reported inability to reliably detect the FOS. Steinbrink et al. failed to detect

a FOS in the visual and somatosensory cortex [41]. Radhakrishnan et al. were unable to detect

the FOS invasively in monkeys; this study differed from other FOS literature due to the invasive

procedure and the use of animal subjects [42]. However, this group reported successful detection of

the FOS in human subjects in a prior study [43]. Consequently, preprocessing plays an important

role in FOS detection. FOS studies have focused on increasing the SNR through trial averaging,

8

band-pass filtering, independent component analysis (ICA) [18,30,36,37], general linear convolution

model [22, 29], frequency-domain analysis [28] and adaptive filtering for pulse correction [23].

A standard practice has yet to be established to measure the FOS. Optical signal intensity and

phase delay measurements have both detected FOS responses, yet there is still debate in literature

as to which is most effective. Frequency domain NIRS systems are advantageous as they allow

simultaneous measurements of intensity and phase delay, as compared to continuous wave systems

which only measure intensity [11,44].

Within the various studies published about the FOS, few mention the potential application for BCIs.

Hu et al. investigated the increase in FOS chaos levels towards BCI classification of tactile stimuli [18].

Baniqued et al. proposed the use of the FOS for BCI error detection and attention assessment [33].

The main challenge towards developing a FOS-based BCI is the low SNR. Previous FOS studies have

averaged a large number of trials, usually above 300, to improve the SNR and thus obtain a reliable

response. However, it has been shown that a reliable response can be detected with 100 trials and

that single trial FOS components have similarities to ERPs measured with EEG [37]. These results

suggest that online classification of the FOS may be possible.

2.4 Automatic Classification of Neuronal Activity

Classification of neuronal signals in BCIs allows to automatically identify neuronal activity corre-

sponding to a mental task or a response to an external stimulus. Using supervised machine learning,

algorithms learn the characteristics of a specific neuronal response through labeled training data.

The trained algorithm can then be used to recognize these response characteristics in new data.

The first step in classifying neuronal signals is preprocessing of the signal to remove other sources of

physiological noise and artifacts. Next, features can be extracted from the preprocessed signal, which

quantify characteristics of the signal in order to differentiate two or more response classes. As BCIs

often collect data from multiple channels and various types of features are extracted, it is common

to obtain large feature sets. With limited BCI training data, large feature sets can lead to overfit-

ting [45]. Therefore, feature selection is used to determine the most discriminative features, which

9

have low variability within each class while separating different classes. The classifier parameters

are then trained with the selected features of the training data. Finally, the trained classifier can be

used to classify new neuronal signal data into the response classes. Ensembles of classifiers can also

be used to improve the classification accuracy by combining the classification decisions of multiple

classifiers.

2.5 Visual Oddball Task and the Prefrontal Cortex Response

Oddball paradigms are commonly used in ERP and FOS studies. Oddball paradigms consist of

sequentially presented auditory or visual stimuli, where oddball stimuli are interspersed amongst

frequent stimuli. Oddball paradigms can be passive, where participants ignore the oddballs, or

active, where participants are instructed to respond to oddballs by pressing a button or keeping a

mental count.

Oddball stimuli have been shown to activate the parietal and prefrontal cortices in fMRI studies [46].

Prefrontal activation, in the middle frontal gyrus (MFG), only occurred if oddballs were considered as

targets, with an active response [47]. ERPs elicited in response to oddballs have also been found in the

parietal and prefrontal cortices. Parietal ERPs include the N200 and P300 while frontal ERPs include

a positive peak around 200 ms (P200) and a negativity between 300 and 500 ms [48,49]. Event-related

oscillations (EROs) have also been reported in response to oddball stimuli, specifically as power

increases in the delta and theta frequency bands for frontal areas [50, 51]. Both visual and auditory

oddball stimuli have been shown to elicit FOS responses in the prefrontal cortex [35, 36, 38, 40].

FOS prefrontal responses to auditory oddball stimuli have been shown to correspond to frontal

ERPs [35].

Using faces, particularly famous faces, as oddball stimuli has been shown to increase the ERP response

by eliciting face-processing related ERPs (N170f and N400) as well as eliciting a larger P300 [52].

This ERP increase was shown to be most significant in the parietal cortex but was also present in

the prefrontal cortex.

10

Chapter 3

Quantifying Fast Optical Signal and

Event-Related Potential Relationships

During a Visual Oddball Task

11

3.1 Abstract

Event-related potentials (ERPs) have previously been used to confirm the existence of the fast optical

signal (FOS) but validation methods have mainly been limited to exploring the temporal correspon-

dence of FOS peaks to those of ERPs. The purpose of this study was to systematically quantify the

relationship between FOS and ERP responses to a visual oddball task in both time and frequency

domains. Near-infrared spectroscopy (NIRS) and electroencephalography (EEG) sensors were co-

located over the prefrontal cortex while participants performed a visual oddball task. Fifteen partici-

pants completed 2 data collection sessions each, where they were instructed to keep a mental count of

oddball images. The oddball condition produced a positive ERP at 200 ms followed by a negativity

300-500 ms after image onset in the frontal electrodes. In contrast to previous FOS studies, a FOS

response was identified only in DC intensity signals and not in phase delay signals. A decrease in DC

intensity was found 150-250 ms after oddball image onset with a 400-trial average in 10 of 15 partici-

pants. The latency of the positive 200 ms ERP and the FOS DC intensity decrease were significantly

correlated for only 6 (out of 15) participants due to the low signal-to-noise ratio of the FOS response.

Coherence values between the FOS and ERP oddball responses were found to be significant in the

3-5 Hz frequency band for 10 participants. A significant Granger causal influence of the ERP on the

FOS oddball response was uncovered in the 2-6 Hz frequency band for 7 participants. Collectively,

our findings suggest that, for a majority of participants, the ERP and the DC intensity signal of

the FOS are spectrally coherent, specifically in narrow frequency bands previously associated with

event-related oscillations in the prefrontal cortex. However, these electro-optical relationships were

only found in a subset of participants. Further research on enhancing the quality of the event-related

FOS signal is required before it can be practically exploited in applications such as brain-computer

interfacing.

12

3.2 Introduction

Near-infrared spectroscopy (NIRS) is a non-invasive imaging tool which can detect neuronal activity

via two types of signals, a slow hemodynamic signal and a fast optical signal. The hemodynamic

signal, corresponding to cerebral blood oxygenation variations, is obtained by measuring absorption

of near-infrared light through extra-cerebral and cerebral tissue. The hemodynamic signal relates to

neuronal activity through neurovascular coupling and has a latency of 4-8 seconds [12]. In contrast,

the fast optical signal (FOS) relates directly to neuronal activity and therefore has a latency on the

order of miliseconds [7].

Based on findings from single studies, the FOS is thought to be caused by changes in optical scattering

properties of cerebral tissue during neuronal activation [53–55]. Stepnoski et al. proposed that the

cause of these scattering changes may be attributable to a variation of the neuronal membrane’s

refractive index during action potential generation [54]. More recently, Lee et al. determined that

the optical response of bulk brain tissue during neuronal activation was related to an increase in

neuronal cell volume [17].

The FOS was first detected non-invasively in the visual cortex of human subjects by Gratton et al.

during a visual stimulation task [21]. Gratton et al. found an increase in relative phase delay of the

FOS following presentation of visual stimuli and labeled this response as the event-related optical

signal (EROS) [21]. This finding of a visually evoked FOS was replicated in subsequent studies by

Gratton et al. [23–28]. The same group also detected the FOS in the auditory [11], somatosensory [56],

motor [20,30–32] and prefrontal [8,33,35] cortices of the brain using phase delay measurements.

The low signal-to-noise ratio (SNR) and high localization of the FOS [25] has led to debate on

FOS detection reliability as well as optimal measurement methods. Using a continuous-wave NIRS

instrument, Steinbrink et al. detected a FOS decrease in intensity during median nerve stimulation

[57]. However, further work in this area revealed that the intensity decrease may have been caused by

motion artifacts rather than a FOS response [41]. Steinbrink et al. were also unable to detect a FOS

in the visual cortex [57]. Using frequency-domain NIRS, Wolf et al. were able to detect a FOS in the

visual cortex with intensity measurements but not with phase delay measurements [58]. Franceschini

13

et al. detected a FOS decrease in intensity during finger tapping but an average of 700-1000 trials was

required for detection [43]. The same group was unable to find a FOS with measurements directly

over the dura mater of monkeys [42].

To ensure a correspondence between the detected FOS response and neuronal activity, the FOS has

been validated spatially with functional magnetic resonance imaging (fMRI) [59] and temporally with

electroencephalography (EEG) measurements of various event-related potentials (ERPs) [33–35,37–

40, 60], including visually-evoked potentials (VEPs) [19, 25, 61, 62]. The temporal correspondence

between FOS and ERP responses in the prefrontal cortex has been verified during Go-NoGo [37],

auditory oddball [33, 35, 38–40, 60] and language processing [34] tasks. However, the FOS oddball

response elicited in the prefrontal cortex during a visual oddball task, where participants keep a

mental count of oddball images [36], has yet to be validated with ERP measurements.

Despite the presence of ERPs in the prefrontal cortex during oddball stimuli, few studies have used

co-localized EEG and NIRS measurements in the prefrontal cortex to explore the correspondence

between oddball elicited FOS and ERP responses. Codispoti et al. found a positive ERP response to

oddball images at 200 ms latency followed by a 300-500 ms negativity in the prefrontal cortex [49].

Similarly, Low et al. found a frontal ERP negativity 300-500 ms after auditory oddball presentation

[35].

Previous studies have mainly focused on the temporal correspondence between FOS and ERP re-

sponses. Following auditory oddball tones, FOS phase delay was found to increase at 350 ms latency,

corresponding to the 300-500 ms latency of an ERP frontal negativity and the P300 latency in

the parietal cortex [35]. Likewise, Medvedev et al. reported a significant correlation between the

200-350 ms latencies of ERP and FOS negative peaks following target images during a Go-NoGo

task [37]. In similar spirit, Tse et al. evaluated the correlation at fixed latencies [63] and the cross-

correlation [40] of linear and quadratic trends of fronto-central FOS and ERP responses during an

auditory oddball task. In both studies, the linear trend of the ERP mismatch negativity response

was found to be correlated with the quadratic trend of the FOS response in the inferior frontal gyrus

(IFG) [40, 63]. Hence, the prefrontal FOS response associated with the visual oddball task has not

been validated using co-localized electrical and optical measurements. Additionally, the spectral

14

relationship between ERP and FOS responses has yet to be quantified.

This present study evaluates the temporal and spectral relationships between FOS and ERP oddball

responses during a visual oddball task using co-localized EEG and NIRS measurements over the

prefrontal cortex. A landmark correlation algorithm was developed to quantify the temporal correla-

tion between FOS and ERP oddball peaks. Spectral relationships were also examined using spectral

coherence and Granger causality connectivity metrics. To our knowledge, this is the first study to

examine the spectral relationships of the FOS and ERP responses during a visual oddball task.

3.3 Methods

3.3.1 Participants

Fifteen able-bodied participants (10 female, mean age: 25.0±3.7 years) were recruited from staff and

students at Holland Bloorview Kids Rehabilitation Hospital (Toronto, Canada) and the University

of Toronto. All participants were right-handed. Participants had normal or corrected-to-normal

vision, were able to read and communicate in English and had no degenerative, cardiovascular or

metabolic disorders and respiratory, psychological, psychiatric or drug and alcohol-related conditions.

Participants were asked to refrain from smoking, and drinking alcoholic or caffeinated beverages

3 hours prior to the data collection sessions. Ethics approval was obtained from Holland Bloorview

Kids Rehabilitation Hospital and the University of Toronto. Written consent was obtained from each

participant.

3.3.2 Instrumentation

NIRS measurements were collected using a frequency-domain near-infrared spectrometer (Imagent

Functional Brain Imaging System from ISS Inc., Champaign, IL) at a sampling rate of 62.5 Hz. Five

laser diode sources, paired at 690 nm and 830 nm wavelengths, and 4 photomultiplier tube detectors

were placed over the participant’s forehead using a custom-made leather headband. A black cloth

was tied over the headband to block external light. The sources and detectors were located at 3 cm

15

distances as shown in Figure 3.1. The source-detector configuration enabled 11 measurement channels

of each wavelength across the prefrontal cortex.

Figure 3.1: NIRS source-detector configuration. Each empty circle, labeled 1-5, represents 2 NIRSsources, one at a wavelength of 690 nm and the other at 830 nm. Each filled circle, labeled A-D,represents a detector. Measurement locations, i.e. channels, are located half-way between sources anddetectors and denoted by a red ’X’. Detector D is on the participant’s left side.

EEG measurements were collected using a BrainAmp DC amplifier (Brain Products GmbH, Ger-

many) at a sampling rate of 250 Hz. Six electrodes were placed around the NIRS headband at

selected International 10-10 system locations with reference electrodes at the mastoids and a ground

electrode at AFz (Figure 3.2). Electrodes were placed on the participant’s skin with doubled-sided

adhesive disks.

Figure 3.2: Location of EEG electrode according to the International 10-10 system. EEG electrodeswere placed at AF3, AF4, F7, F8, F9 and F10 electrode locations shown in green. The groundelectrode, shown in purple, was placed at AFz and the reference electrodes, shown in orange, wereplaced at the left and right mastoids.

16

3.3.3 Task Paradigm

The visual oddball task consisted of images appearing one at a time in the centre of a computer

screen with 2-6 frequent images appearing between each oddball image. The oddball image was an

image of Einstein’s face and the frequent images were alphabetical letters A, B, C and D. Within

a session, 20% of images were oddballs. Images appeared at a frequency of 1.9531 Hz (i.e. every

512 ms). This stimulus frequency ensured that image presentation onset was congruent with the

62.5 Hz NIRS sampling rate (i.e. each stimulus appeared at the start of the NIRS source cycle).

This stimulus frequency facilitated observation of the FOS response with non-overlapping 500 ms

epochs. Participants were instructed to keep a mental count of oddball images to increase signal

activation [35,36].

3.3.4 Experimental Protocol

Each participant attended 2 data collection sessions, which took place on different days separated

by at most 30 days. During the sessions, participants were seated in a dimly lit room facing a

computer screen, on which the task paradigm was displayed. Each session was composed of 3 runs.

Each run began with a 30 second baseline period, during which a fixation cross was presented on

the computer screen, followed by 10 blocks of the visual oddball task. Each block contained 100

trials of the visual oddball task, where a trial was defined as the presentation of one image. After

each block, participants were asked to enter the number of oddballs counted during the block. The

experimental protocol for each session is shown in Figure 3.3. NIRS and EEG measurements were

collected simultaneously in both sessions.

The rest period between blocks and runs was participant-determined; participants clicked on a button

on the computer screen to start the next block or the next run when they were ready to proceed.

Participants were instructed to avoid moving during the baseline period or during the blocks but they

could move and re-position themselves between blocks. Each session was approximately 90 minutes

including setup and data collection.

17

Figure 3.3: Experimental protocol for one session. The oddball image of Einstein’s face was replacedby a smiley face in this figure for copyright purposes.

3.3.5 Data Analysis

3.3.5.1 FOS Preprocessing

Phase delay and DC intensity FOS measurements from 690 nm and 830 nm wavelength sources

were analyzed. Phase data were obtained using a fast Fourier transform (FFT), corrected for phase

wrapping and referenced to the respective 690 nm and 830 nm medial channel C3, shown by detector

C and source 3 in Figure 3.1, to obtain relative phase delay data.

Motion artifacts were removed from DC intensity signals using independent component analysis

(ICA). As similar artifacts were present in both 690 nm and 830 nm DC signals, ICA was performed

on a combined array of all 690 nm and 830 nm wavelength channels (a total of 22 channels) us-

ing the FastICA algorithm [64], resulting in 22 independent components. To identify independent

components that represented artifacts, we invoked a wavelet-based screening procedure [65]. Each

independent component was subjected to a 5-level decomposition using Daubechies order 4 wavelets.

Detail coefficients at levels 3 and 5 (corresponding to 7.8-15.6 Hz and 2.0-3.9 Hz, respectively) were

found to capture artifact peaks. Thresholds for d3 and d5 coefficients were set empirically through

visual inspection to 4 times the standard deviation of the respective coefficients. Independent com-

ponents whose d3 or d5 coefficients contained a peak above the respective threshold were removed.

Up to 3 artifact components were removed when reconstructing the DC intensity signals from the

18

independent components. DC intensity signals were then re-grouped into 690 nm and 830 nm DC

intensity arrays with 11 channels per array.

Next, phase delay and DC intensity data were linearly detrended, then band-pass filtered between 2

and 20 Hz with a 9th order Chebychev filter to remove low-frequency hemodynamic signal components

and physiological noise, such as heart pulse, respiration and the Mayer wave, as well as high-frequency

noise. Remaining heart pulse artifacts were removed with the heart pulse filter developed by Gratton

et al. [23]. DC intensity signals were standardized (i.e. zero mean and unit standard deviation) per

block. Phase delay signals were normalized per block by subtracting the mean of the block. DC

intensity and phase delay signals were segmented into 500 ms trial epochs time-locked to stimulus

presentation. The epoched 500 ms trials were then baseline-corrected with the 100 ms pre-stimulus

signal. The first trial epoch of each block was eliminated to avoid artifacts present at the onset

of NIRS data collection. Four hundred trials were then averaged per stimulus type (oddball and

frequent) to increase SNR. Figure 3.4 illustrates the preprocessing steps.

Figure 3.4: FOS preprocessing steps for DC and phase delay measurements.

3.3.5.2 ERP Preprocessing

EEG data, automatically referenced to the right mastoid, were re-referenced with the left mastoid

electrode and band-pass filtered between 0.5 and 30 Hz. Ocular artifacts were removed using a

modified version of the ADJUST algorithm [66]. The ADJUST algorithm was modified by utilizing

only the frontal spatial electrode configuration and adjusting artifact thresholds to ensure that all

ocular artifacts were removed. The preprocessed EEG data were then segmented into 500 ms epochs

time-locked to stimulus presentation and baseline-corrected with the 100 ms pre-stimulussignal. EEG

preprocessing was performed using the EEGLAB Matlab toolbox [67].

19

3.3.5.3 FOS-ERP Temporal Landmark Correlations

A landmark correlation algorithm was developed to evaluate the correlation between FOS and ERP

peak latencies. To obtain FOS and ERP averaged responses, 400 oddball trials were randomly

sampled and averaged per session and participant. The same trials were averaged for both modalities.

The ERP average was downsampled to the FOS sampling frequency. No information was lost from

downsampling the ERP average as the ERPs were bandpass filtered below the Nyquist frequency

of the FOS. For each session and participant, the 2.5 and 97.5 percentiles of the ERP amplitude

distribution over the 400-oddball average were used as the negative and positive ERP peak thresholds,

respectively. Positive ERP peaks were identified within 150-250 ms latencies from image onset and

negative ERP peaks within 275-400 ms latencies. These time windows were selected corresponding

to frontal ERP latencies reported in previous literature [35,49].

The 2.5 and 97.5 percentiles of the FOS average amplitudes were used as the negative and positive

FOS peak thresholds, respectively. FOS peak latencies were identified across the 500 ms epochs.

FOS-ERP correlations were evaluated with a triangular function, where FOS and ERP peaks of

equal latency had a correlation value of 1 and FOS peak latencies ±100 ms away from the ERP peak

latency had a correlation of 0. The correlation value decreased linearly from 1 to 0 between peak

distances of 0 to 100 ms, respectively.

The significance of the correlations was determined with a permutation test (p < 0.05), where FOS

trial labels where permuted 1000 times to obtain a null distribution of no correlation. Using the

landmark correlation algorithm, the correlation between peak latencies of the average FOS-permuted

trials and those of the average oddball ERP was calculated in each run of the permutation test. For

a given ERP-FOS channel pair, a significant landmark correlation in the neighbourhood of an ERP

oddball peak was then identified as that exceeding the 95th percentile of the landmark correlations

from the permutation test.

20

3.3.5.4 FOS-ERP Spectral Relationships

Coherence and spectral Granger causality connectivity metrics were calculated to evaluate the spec-

tral relationships between FOS and ERP responses to oddball images. These spectral relationships

were separately computed using the FieldTrip Matlab toolbox [68]. Coherence and spectral Granger

causality were calculated for all FOS-ERP channel pairs across 400 randomly selected oddball trials.

ERP trials were downsampled to the FOS sampling frequency before computing the relationships.

Coherence Cfe was calculated as

Cfe(w) =|Pfe(w)|2

Pff (w)Pee(w)(3.1)

where Pfe is the cross power spectral density of corresponding FOS and ERP trials, whereas Pff and

Pee are the power spectral densities at frequency w of FOS and ERP trials, respectively. The spectral

Granger causality was evaluated according to the Geweke spectral measure as described in [?]. The

Geweke spectral measure is derived from the Fourier transform of a bivariate autoregressive model,

which provides the spectral density matrix

S(w) = H(w)ΣH∗(w) (3.2)

where Σ is the covariance matrix, H(w) is the transfer function matrix and the asterisk (*) denotes

the complex conjugate. The Granger causality from ERP response to FOS responses at frequency w

was therefore defined as

fERP→FOS(w) = lnSff (w)

Σ(fe)ff |H

(fe)ff (w)|

2 . (3.3)

The significance of the spectral coherence between ERP and FOS responses was determined with

a permutation test (p < 0.01). Spectral coherence was calculated 1000 times with permuted FOS

labels to obtain a null distribution of spectral coherence for each channel pair and frequency bin. The

21

99th percentiles were found for all null distributions of spectral coherence for each channel pair and

frequency bin. To correct for multiple comparisons, the largest 99th percentile value was selected as

the critical value for determining significant spectral coherence [69]. The same procedure was used

to test for significant values of Granger causality between ERP to FOS responses.

3.4 Results

3.4.1 ERP Response

The ERP 100-trial average in Figure 3.5 demonstrates a positive peak near 200 ms followed by

a negativity 300-500 ms after image presentation for oddball trials. This ERP oddball response

was identified in 12 of 15 participants. For most participants the response was consistent across all 6

prefrontal electrodes. Previous ERP studies have reported a corresponding prefrontal negativity 300-

500 ms after oddball presentation [35, 49]. Codispoti et al. also reported a positive ERP differential

response to rare targets at a latency of 200 ms [49].

Figure 3.5: ERP 100-trial average response for participant 7. The red line denotes the oddball imageresponse and the blue line denotes the frequent image response. Time 0 denotes the instance of imagepresentation.

22

3.4.2 FOS Response

When averaging 400 FOS trials, DC intensity was found to decrease between 150 and 250 ms following

oddball image presentation, similarly to the findings of [36]. A decrease in DC intensity between 150

and 250 ms was found across 2 sessions in 10 participants. A similar decrease was found in 690 nm

and 830 nm DC intensity signals, however due to the low SNR, the responses were not present in

both wavelengths for the 10 participants. The FOS response to oddball images occurred in the

left prefrontal cortex channels and also in the furthest right channels for some participants. An

example of the 400-trial average FOS response across sessions 1 and 2 is shown in Figure 3.6. For

some participants, the 150-250 ms negativity was followed by a positive peak around 300 ms. Due

to the low SNR, negative peaks in the 150-250 ms range were not always identified as they fell

below the FOS peak threshold. Notably, a FOS response was not found in the relative phase delay

measurements.

Figure 3.6: FOS 400-trial averaged response for channel B3 of participant 14. The red lines denotethe oddball image response, the blue lines the frequent image response. The asterisk denotes a FOSpeak that exceeded the FOS peak threshold, meaning that the FOS peak was significantly correlatedtemporally with a corresponding ERP peak.

23

3.4.3 FOS-ERP Temporal Landmark Correlations

Non-zero correlations between FOS and ERP oddball peaks were identified in 9 participants; however,

statistically significant correlations were found in only 7 participants (permutation test, p < 0.05).

FOS negative peaks around 200 ms were significantly correlated with positive ERP peaks for 6

participants. For 1 participant, a significant correlation was also found between the negative ERP at

300 ms and a negative FOS peak around 275 ms. Figure 3.7 depicts a significant correlation between

FOS and ERP responses for one participant.

Figure 3.7: DC 830 nm FOS-ERP correlation in session 1 of participant 6. The top plot shows theFOS DC intensity oddball response for channel B5 and the bottom graph the ERP oddball responsefor channel AF3. The shaded vertical bar denotes the latency between the significantly correlated FOSand ERP peaks (p < 0.05).

24

3.4.4 FOS-ERP Spectral Relationships

The coherence of FOS and ERP oddball responses were found to be statistically significant (permuta-

tion test, p < 0.01) in the 3-5 Hz frequency range. DC intensity measurement at 830 nm, for channels

D4 and A2, were significantly correlated with all ERP channels in terms of coherence in both sessions

for 8 participants and in one session for 2 participants. Coherence (permutation test, p < 0.01) was

also found in the 3-5 Hz frequency range between DC intensity of 690 nm wavelength FOS channels

and ERP channels in both sessions of 5 participants and in one session for 6 participants. Significant

coherence between DC intensity 690 nm and ERP was found across FOS channels D4, D5, C4, A2

and A3. Figure 3.8 shows an example of the coherence values for all DC 830 nm FOS and ERP

channel pairs in the 2-20 Hz frequency range.

Figure 3.8: DC 830 nm FOS-ERP coherence for session 2 of participant 4. (a) FOS-ERP coherencevalues. (b) FOS-ERP channel pairs with statistically significant coherence (p < 0.01) are shown inred. Significant coherence is shown for FOS channels A2, A3, B5 and D4.

ERP responses to oddball images were found to have a Granger causal influence in the frequency

domain on FOS oddball responses. Significant Granger causal influences (permutation test, p < 0.01)

of ERP channels (2-6 Hz) on FOS DC intensity 830 nm channels D4, D5 and A2 were found in both

sessions for 3 participants and in one session for 4 participants. ERP channels also had a Granger

causal influence on FOS DC intensity 690 nm channels D4, D5 and A3 in the 2-5 Hz range in

both sessions for 4 participants and in one session for 3 participants. Frequency-domain Granger

causalities from ERP to DC 830 nm FOS channels are shown for one participant in Figure 3.9.

Significant Granger causal influences from FOS to ERP channels were not found to be consistent

across participants or frequency bins.

25

Figure 3.9: ERP to FOS (DC 830 nm) Granger causality for session 2 of participant 5. (a) ERPto FOS Granger causality values. (b) FOS-ERP channel pairs with statistically significant Grangercausal influences of ERP on FOS are shown in red. Significant Granger causality is shown betweenall ERP channels and FOS channels A2, B5 and D5.

3.5 Discussion

This study quantified the correlation between FOS and ERP responses in the prefrontal cortex

during a visual oddball task. ERPs were used to validate the presence of a neuronal response in

the prefrontal cortex during the visual oddball task. Oddballs were found to elicit a positive ERP

peak around 200 ms and an ERP negativity 300-500 ms after stimulus onset, as found in [35,49,70].

Keeping a mental count of oddballs during a visual oddball task has been shown to elicit activation

in the prefrontal and parietal areas of the brain in fMRI and EEG studies, specifically in the middle

frontal gyrus (MFG) [47]. Huettel et al. proposed that oddballs elicit prefrontal activity as they

require a change in response strategy compared to frequent stimuli [46]. Only task-related oddballs

have elicited a frontal positive ERP at 200 ms and the frontal ERP response has been shown to

be independent of the task (mental counting or button-press), which suggests that the frontal ERP

oddball response is related to stimulus evaluation [48].

FOS DC intensity was found to decrease between 150 and 250 ms after oddball image onset, as

reported by [36]. Although the FOS response was more clearly identified in one of the wavelengths

than the other for most participants, a similar decrease was found in both 690 nm and 830 nm

wavelength signals. As found in previous FOS literature [25], a FOS response of the same sign in 690

nm and 830 nm wavelengths confirms that the FOS is due to near-infrared light scattering in cerebral

26

tissue. If the response were caused by near-infrared light absorption due to rapid deoxygenation,

690 nm and 830 nm response would be of opposite signs as these wavelengths are on either side

of the hemoglobin isosbestic point, where the absorption spectra of oxygenated and deoxygenated

hemoglobin intersect.

Large variations were present in FOS responses to oddball and frequent images even when averaging

400 trials. Attenuation of the DC intensity of the FOS at 150-250 ms was temporally correlated with

the ERP 200 ms positive peak and 300 ms negativity for 10 participants. However, when evaluating

the significance of these temporal correlations with a permutation test, only 7 participants were found

to have statistically significant FOS-ERP temporal correlations. The significant temporal correlation

between the decrease in FOS DC intensity and the electrical EEG peaks validated that the decrease

in the FOS DC intensity was related to neuronal activity. The FOS-ERP temporal correlations

confirmed the correspondence between electrical and optical measurements of neuronal activity as

found in previous literature [35,37].

ERPs have been shown to be associated with a frontal power increase in theta band (4-7 Hz) and

a parietal delta band (0-3 Hz) increase [51]. Jones et al. reported that event-related oscillations for

the oddball target condition had a maximum upper delta and theta band (3-7 Hz) power in frontal

electrodes [50]. Additionally, the power of frontal theta oscillations was correlated with the P200

ERP amplitude at the Fz electrode [50]. These findings are in agreement with the results of the

present study, which demonstrated a positive ERP peak at 200 ms and upper delta and theta band

(2-6 Hz) spectral relationships between the FOS and ERP oddball responses.

Coherence of FOS and ERP oddball responses was significant in the 3-5 Hz range for both 690 nm

and 830 nm FOS signals of 10 participants. A Granger causal influence of ERP on FOS was also

found in the 2-6 Hz range in 690 nm and 830 nm signals of 7 participants. Spectral relationships

were found in the furthest left and right FOS channels. Frequency-domain relationships were more

consistent across sessions and participants than temporal correlations. This suggests that spectral

measures may enable more reliable FOS detection than previously reported temporal measures.

A FOS response was not found in phase delay measurements. Previous studies have also reported

a FOS response with DC intensity but not with phase delay measurements [30, 58]. However, this

27

contradicts the results of previous FOS studies which demonstrated that the FOS was best measured

with phase delay [20,25,29]. The lack of a FOS response in phase delay measurements may, in part,

have been due to instrument limitations. A single reference channel was used to obtain relative phase

delay measurements across all channels. An NIRS instrument with more sources and detectors may

allow for co-localized phase delay referencing. However, it is possible that phase delay measurements

do not reveal a FOS response.

This study included the measurement of FOS response to oddballs only in the prefrontal cortex.

Although FOS studies have focused on oddball response in the prefrontal cortex, previous ERP

studies for oddball paradigms have focused on parietal and central brain regions. The amplitude of

ERPs has been shown to be larger in parietal and central regions than in the prefrontal cortex, where

prefrontal response tend to be associated with rare or novel stimuli [70,71]. Additional measurement

channels over the central and parietal regions may allow detection of a FOS response with a larger

SNR.

Additional measurement channels would also facilitate the use of noise removal techniques such as

ICA, which has been shown to improve the SNR of the FOS in previous studies [30, 36]. ICA was

used in this study to remove large artifacts in the raw NIRS signal. However, a larger number

of channels or multiple co-localized channels would have been required to effectively remove noise-

related components from the FOS. Further channels would allow separation into a greater number of

independent components, thus reducing the risk of obtaining components containing a combination

of noise and the FOS response.

3.6 Conclusion

This study performed co-localized NIRS and EEG measurements over the prefrontal cortex of 15

participants during a visual oddball task. A FOS response to oddballs was identified in 10 of 15

participants using DC intensity measurements of the NIRS signal. A FOS response was not found

using phase delay measurements. Significant correlations were identified in the time and frequency

domains between FOS DC intensity and ERP responses to visual oddball stimuli. However, spectral

28

relationships were shown to have greater consistency across sessions and participants. Coherence and

Granger causality revealed FOS-ERP oddball response relationships in the upper delta and theta

frequency bands, which coincide with previously reported event-related oscillations in the prefrontal

cortex.

29

Chapter 4

Online Classification of the Near-Infrared

Spectroscopy Fast Optical Signal for

Brain-Computer Interfaces

30

4.1 Abstract

The fast optical signal (FOS), measured with near-infrared spectroscopy (NIRS), has high temporal

and competitive spatial resolution which provides an opportunity for a novel brain-computer interface

modality. However, the reliability of the FOS has been debated due to its low signal-to-noise ratio.

This study examined the feasibility of automatically classifying the prefrontal FOS response during

a visual oddball task. FOS measurements were collected from 15 participants during 3 offline and

2 online sessions. Classification feedback was provided to participants during the online sessions.

The FOS classification algorithm discriminated between oddball and frequent responses. Separate

classifiers were created for DC intensity and phase delay FOS measurements. The decisions of these

classifiers were combined with a weighted majority vote. Fifteen-trial averages were selected for

optimal classification results and the best feature types were found to be the number of zero crossing

and the variance. FOS responses to oddball and frequent images were classified offline with an average

balanced accuracy of 62 ± 5% and classified online with an averaged balanced accuracy of 63 ± 6%

across all participants. Offline classification accuracies were significantly higher than chance for all

participants. Online classification results were significantly higher than chance in both online sessions

for 7 of 14 participants. ERPs were also classified using a similar algorithm with an average balanced

accuracy of 77 ± 5%, which confirmed that the prefrontal neuronal response to the visual oddball

task could be classified above the level required (> 70%) for effective BCI communication. The

FOS classification results demonstrated that automatic classification of the FOS is possible at above-

chance levels, however, FOS classification accuracies did not reach the effective BCI communication

threshold. Further FOS classification efforts should focus on investigating spectral features as well

as adding measurement channels over the fronto-central and parietal areas of the brain.

31

4.2 Introduction

Brain-computer interfaces (BCIs) enable control of an external device by decoding cognitive activity.

BCIs can therefore provide a method of communication for individuals with severe motor impairment

as no movement or speech is required. Near-infrared spectroscopy (NIRS) is an optical BCI modality

which has increased in popularity in recent years due to its ease of use, portability, relatively low

cost and high spatial resolution, on the order of 1 cm [5]. NIRS is commonly used to detect oxygen

level variations in the cerebral cortex, known as the hemodynamic signal, by measuring near-infrared

photon absorption through extra-cerebral and cerebral tissue. The hemodynamic signal relates to

neuronal activity through neurovascular coupling. In particular, neuronal activation increases cere-

bral blood flow and oxygen absorption in cerebral tissues in the activated region of the brain [13].

The temporal resolution of the hemodynamic response is 4-8 seconds [12]. As current NIRS BCIs

measure the hemodynamic signal, the information transfer rate of NIRS BCIs is limited due to the

hemodynamic response latency [12].

NIRS can also measure a fast optical signal (FOS) related to changes in near-infrared photon scatter-

ing through cerebral tissue during neuronal activation. These optical scattering variations in cerebral

tissue are thought to be caused by cellular volume and ion concentration fluctuations during neuronal

activation [17]. As the FOS directly relates to neuronal activation, the latency of the FOS response

is on the order of 100 ms [7]. The FOS, also known as the event-related optical signal (EROS), was

first detected in human subjects by Gratton et al. in 1995 [20, 21]. Since then, the FOS has been

most studied in the visual [19, 21–28] and prefrontal cortices [8, 33–40] of the brain, but it has also

been detected in the motor [20,29–32], somatosensory [18] and auditory [11] cortices.

FOS activity has been detected in the prefrontal cortex using various activation tasks and stimuli.

During visual and auditory oddball target detection, inhibitory processes (No-Go) were found to

increase FOS activity near the border between the middle frontal gyrus (MFG) and inferior frontal

gyrus (IFG), whereas keeping a mental count of oddball targets increased FOS activity in the MFG

[35]. During a Go-NoGo paradigm, target detection was found to increase FOS activity in the right

MFG and left IFG [37]. Task switching, from a verbal task to a spatial task and vice versa, was found

to increase FOS activity in the left IFG and MFG [8]. Language processing was also found to activate

32

a FOS response in the left IFG [34]. FOS response latencies in the prefrontal cortex ranged from

100-350 ms [8, 33–37, 39]. Although the FOS has higher temporal resolution than the hemodynamic

signal, the FOS has a low signal-to-noise ratio (SNR). The FOS amplitude in the prefrontal cortex

has ranged between 0.01-0.05% for relative changes in intensity (∆I/Io) [36] and between 0.03-0.2◦

for phase delay measurements [35,39].

The reliability of the FOS has been disputed due to its low SNR. In addition, the high localization of

the FOS increases the difficulty in obtaining a reliable response [29]. Steinbrink et al. failed to detect

a FOS in the visual and somatosensory cortices [41]. Radhakrishnan et al. were unable to detect the

FOS directly over the dura mater of monkeys, a study which differed from other FOS literature due

to the invasive procedure and the use of animal subjects [42]. However, the same group had reported

successful detection of the FOS in human subjects in a prior study [43].

A standard practice has yet to be established for FOS measurement and analysis. The FOS has

been detected with both frequency-domain NIRS systems, which can measure both intensity and

phase delay, and continuous-wave NIRS systems, which only measure intensity [11]. There is debate

in the literature regarding whether intensity or phase delay measurements are most effective for

FOS detection. Due to the low SNR, preprocessing plays an important role in FOS detection. FOS

studies have used various methods to improve the SNR, including trial averaging, band-pass filtering,

independent component analysis (ICA) [18, 30, 36, 37], general linear convolution models [22, 29],

frequency-domain analysis [28] and adaptive filtering for pulse correction [23]. Previous FOS studies

have averaged a large number of trials, ranging between 100 [37] and 3000 [41], to improve the

SNR.

Possessing high temporal and spatial resolution, the FOS has largely featured in functional connec-

tivity studies, with few even acknowledging the potential as a novel BCI modality. Hu et al. used an

increase in chaos levels corresponding to a FOS tactile stimulus response to differentiate periods of

tactile stimulus from rest [18]. Baniqued et al. proposed the use of the FOS for BCI error detection

and attention assessment [33]. However, the FOS has yet to be automatically classified and hence

its value as a control signal for BCIs remains unknown.

33

In this study, simultaneous NIRS and EEG measurements were collected during a visual oddball

task. EEG measurements of ERPs were collected to confirm that the prefrontal neuronal response to

a visual oddball task could be classified. Several studies have used ERPs to validate FOS responses

in the prefrontal cortex [35, 37, 40, 60]. An image of Einstein’s face was used for the oddball image

as famous faces are known to increase P300 amplitudes and to elicit additional ERPs related to face

perception, known as the N170 and N400f, occurring at 130 to 200 ms and 300 to 400 ms latencies

respectively [52]. Face perception ERPs have been mainly elicited in centro-parietal electrodes but

have also been visible in frontal electrodes for famous face stimuli [52].

Offline analysis was performed to determine the required multiplicity of trials and duration of ob-

servation for FOS feature extraction to yield an algorithm for subsequent online classification of the

FOS response. DC intensity and phase delay data were used to classify the FOS response to oddball

and frequent images with a weighted majority vote of support vector machine (SVM) and linear

discriminant analysis (LDA) classifiers. FOS responses to a visual oddball task were classified online

to provide participants with binary feedback relating to classification decisions. To our knowledge,

this is the first study to automatically classify the FOS response online in a NIRS BCI context.

4.3 Methods

4.3.1 Participants

Fifteen right-handed adult participants with a mean age of 25.0 ± 3.7 years took part in the study.

Participants had normal or corrected-to-normal vision, could communicate in English and did not

have any health conditions which may have affected NIRS measurements, such as cardiovascular,

respiratory or psychological conditions. The study protocol was approved by the research ethics

board at Holland Bloorview Kids Rehabilitiation Hospital and the University of Toronto. Informed

written consent was obtained from participants prior to the first data collection session.

34

4.3.2 Instrumentation

NIRS measurements were recorded over the prefrontal cortex with an Imagent frequency-domain

spectrometer (ISS Inc., Champaign, IL) at a sampling rate of 62.5 Hz. Laser diode sources and pho-

tomultiplier tube detectors were placed in a custom leather headband on the participant’s forehead.

Five paired 690 nm and 830 nm wavelength sources and 4 detectors at 3 cm distances enabled 11 mea-

surement channels of each wavelength across the prefrontal cortex (shown as red X’s in Figure 4.1).

External light was blocked by placing a black cloth over the headband. NIRS sources were sampled

in a 8-source cycle with a 2 ms “on” period per source. Sources 2 and 5 shared the same on period

as they were furthest apart. Sources were modulated at 110 MHz with a cross-correlation frequency

of 5 kHz. Detectors were sampled at 20 kHz, providing 10 waveforms per source “on” period.

EEG measurements were recorded at a sampling rate of 250 Hz with a Brain Amp DC amplifier

(Brain Products GmbH, Germany). Electrodes were placed around the NIRS headband at AF3,

AF4, F7, F8, F9 and F10 International 10-10 system locations. The ground electrode was placed at

AFz and the reference electrodes at the mastoids. Double-sided adhesive disks (Brain Vision LLC)

were used to secure electrodes to the participant’s skin.

Figure 4.1: NIRS and EEG channel configuration. White circles denote paired 690 nm and 830 nmwavelength NIR sources, black circles denote detectors and red X’s denote measurement channels.Grey diamonds denote EEG measurement electrode locations and the red diamond denotes the groundelectrode location.

35

4.3.3 Experimental Protocol

4.3.3.1 Offline

Participants took part in 3 offline data collection sessions during which they performed a visual

oddball task. Sessions were on different days, with a maximum of 90 days between sessions. NIRS

and EEG measurements were collected simultaneously in sessions 1 and 2, while in session 3 only

NIRS measurements were collected to obtain additional training data for classification of the FOS

response. Each session consisted of 3 runs, with a 30 s baseline period and 10 visual oddball task

blocks per run, as shown in Figure 4.2.

During a visual oddball task block, 100 images were presented sequentially on a computer screen

for 512 ms each. Oddball images (Einstein) were presented semi-randomly with 2-6 frequent images

(alphabetical letters A, B, C, D) between two successive oddball images. Throughout the session, 20%

of images were oddballs. Participants were instructed to keep a mental count of oddball images.

Further detail about the task paradigm and experimental protocol can be found in [72], where data

from sessions 1 and 2 of each participant were previously used in the analysis of the temporal and

spectral relationships between FOS and ERP responses.

Figure 4.2: Data collection session protocol

36

4.3.3.2 Online

Fourteen of the 15 participants who completed the offline sessions also attended 2 online sessions on

separate days. One participant did not take part in the online sessions due to lack of availability. Each

online session was composed of 1 offline run, to collect same-session training data, and 2 online runs.

Feature selection and classifier training were performed during the session, following the first run.

Each run was composed of a 30 s baseline period and 10 blocks of the visual oddball task. Each block

contained 100 trials, where 15-25 trials per block were oddball images. Only NIRS measurements

were collected during the online sessions.

During the 2 online runs, binary feedback was provided to the participants after each block. Feedback

indicated whether the oddball target image was correctly identified as shown in Figure 4.3. An overall

classification score was also provided as part of the feedback.

Figure 4.3: Online feedback presented after each block during the online runs for (a) successfully and(b) unsuccessfully classified blocks.

4.3.4 Offline FOS Analysis

Offline FOS classification was conducted to confirm the feasibility of automatic FOS classification as

well as to optimize classification algorithm parameters.

4.3.4.1 Preprocessing

Both DC intensity and phase delay data were extracted from NIRS signals using a methodology

similar to that of [25]. The first and last waveforms of each source ’on’ period were removed to avoid

cross-talk between sources. For each source “on” period, DC intensity was calculated by averaging

37

the 8 remaining waveforms and phase delay was calculated as the unwrapped angle of a fast Fourier

transform (FFT). Therefore, DC intensity and phase delay were extracted from individual sources

at a sampling rate of 62.5 Hz (16 ms). Phase delay signals of each wavelength were referenced to the

medial channel C3 to obtain relative phase delay.

Motion artifacts were removed from DC intensity data using independent component analysis (ICA)

as described in [72]. Phase data were corrected for phase wrapping and referenced to the medial

NIRS channel of the respective wavelength. Following linear detrending, a 9th order Chebychev

bandpass filter with a passband of 2 to 20 Hz was applied to both DC intensity and phase delay

data in order to remove low-frequency hemodynamic signal components and physiological noise (i.e.

heart pulse, respiration and the Mayer wave), in addition to high-frequency noise. Remaining heart

pulse artifacts in the DC intensity data were removed using the adaptive pulse filter from Gratton

et al. [23].

DC intensity channel-specific data were then standardized per block by subtracting the block mean

and dividing by the block standard deviation, while phase delay data were normalized per block by

subtracting the block mean. DC intensity and phase delay data were segmented into 500 ms trial

epochs time-locked to stimulus onset and baseline-corrected with a 100 ms pre-stimulus mean. The

first trial epoch per block was removed due to NIRS data collection onset artifacts.

4.3.4.2 Feature Extraction

Temporal and spectral features were extracted from stimulus type (oddball or frequent) trial averages

of DC intensity and phase delay trial epochs (Table 4.1). Sequential trials of each stimulus type were

averaged across each run. As the number of oddballs varied per block, remaining trials at the end of

each run were not used. Trial averaging was varied between 5 and 25 trials for feature extraction to

determine the optimal trial-average that improved the SNR while maintaining a sufficient number of

training data.

Features were extracted from individual channels as well as from channel combinations. Neighbouring

channels [A1, A2, A3], [A1, A2, A3, C1], [B3, B4, B5], [D4, D5], [B3, B4, B5, D4, D5] and [B5, D4,

38

D5] were averaged to improve the SNR without requiring further trial averaging. Right-side channels

were subtracted from corresponding left-side channels (i.e. B4-A1, B5-A2 and B3-A3) to obtain

features related to a lateralized FOS response. Features were therefore extracted from 6 channel

averages, 3 lateral channel differences and all individual channels (10 for phase delay and 11 for DC

intensity). Phase delay features were extracted from 10 individual channels as the medial channel

C3 was used as a reference.

To obtain spectral features, the power spectral density was calculated for each trial using a FFT

and then averaged across trials. Extracted spectral features included relative power spectral density

(PSD) for 2-4 Hz, 5-8 Hz, 9-12 Hz and 13-18 Hz frequency bands, corresponding to the frequency

ranges for EEG delta, theta, alpha and beta bands respectively.

Prior to extracting temporal features, trials were averaged and a 3-point moving average was applied

to smooth the trial averages. Temporal features were extracted over a 0-500 ms time window to

capture the FOS response up to 500 ms following the onset of image presentation. Temporal features

included amplitude and latency of the largest positive peak and largest negative peak, which were

set to zero if no peak amplitude exceeded the standard deviation of the time window. The amplitude

difference, latency and slope between the largest positive and negative peaks, the number of zero

crossings and the largest Lyapunov exponent (LLE) were also extracted from the 0-500 ms time

window. The largest Lyapunov exponent is a measure of chaos in dynamical systems and has been

previously touted as being sensitive to FOS responses to tactile stimuli [18].

Additional temporal features were extracted over the 0-500 ms time window as well as 96-192 ms,

192-304 ms, 272-400 ms time windows, corresponding to the latency of frontal ERPs [35, 49, 52] as

well as DC intensity and phase delay FOS oddball response peaks [35–37]. These features included

latency and amplitude of the maximum and minimum, kurtosis, variance, and positive and negative

area of FOS responses in each time window.

All the features were extracted from the 4 measurement types, DC intensity and phase delay at

690 nm and 830 nm wavelengths, to obtain 4 feature sets. DC intensity feature sets contained a

total of 900 features each, including 80 spectral features (20 channel combinations × 4 frequency

bands) and 820 temporal features (20 channel combinations × (1 time window × 9 features types

39

+ 4 time windows × 8 feature types)). Phase delay feature sets contained a total of 855 features

each, including 76 spectral features (19 channels × 4 frequency bands) and 779 temporal features (19

channel combinations × (1 time window × 9 features types + 4 time windows × 8 feature types)).

The feature types extracted from each channel combination are summarized and numerically indexed

in Table 4.1.

Table 4.1: Feature types extracted from all individual channels and channel combinations of DCintensity and phase delay at 690 nm and 830 nm (P2P = peak to peak, + = positive, − = negative,LLE = largest Lyapunov exponent, PSD = power spectral density).# Feature Type # Feature Type # Feature Type1 + Peak latency (0-500 ms) 16 Max amp. (192-304 ms) 31 Variance (96-192 ms)2 + Peak amp. (0-500 ms) 17 Max amp. (272-400 ms) 32 Variance (192-304 ms)3 − Peak latency (0-500 ms) 18 Min latency (0-500 ms) 33 Variance (272-400 ms)4 − Peak amp. (0-500 ms) 19 Min latency (96-192 ms) 34 + Area (0-500 ms)5 P2P amp. (0-500 ms) 20 Min latency (192-304 ms) 35 + Area (96-192 ms)6 P2P latency (0-500 ms) 21 Min latency (272-400 ms) 36 + Area (192-304 ms)7 P2P slope (0-500 ms) 22 Min amp. (0-500 ms) 37 + Area (272-400 ms)8 # zero crossings (0-500 ms) 23 Min amp. (96-192 ms) 38 − Area (0-500 ms)9 LLE (0-500 ms) 24 Min amp. (192-304 ms) 39 − Area (96-192 ms)10 Max latency (0-500 ms) 25 Min latency (272-400 ms) 40 −Area (192-304 ms)11 Max latency (96-192 ms) 26 Kurtosis (0-500 ms) 41 − Area (272-400 ms)12 Max latency (192-304 ms) 27 Kurtosis (96-192 ms) 42 2-4 Hz PSD13 Max latency (272-400 ms) 28 Kurtosis (192-304 ms) 43 5-8 Hz PSD14 Max amp. (0-500 ms) 29 Kurtosis (272-400 ms) 44 9-12 Hz PSD15 Max amp. (96-192 ms) 30 Variance (0-500 ms) 45 13-18 Hz PSD

4.3.4.3 Feature Selection and Classification

Feature selection and offline classification were performed on feature sets for the 4 measurement

types and an additional feature set which combined the features from all measurement types (3510

features). Training and testing feature values were normalized using the minimum and maximum

values of each feature within the training set such that all training features ranged between 0 and 1.

A fast correlation-based filter (FCBF) was applied to the training data to select the top features of

each set. The number of selected features was varied between 5 and 25 to determine the optimal

number for classification.

SVM and LDA classifiers were trained with each reduced feature set (a total of 5 feature sets), for

a total of 10 classifiers. A radial basis function (RBF) kernel was used for the SVM classifiers and

40

an SVM extension based on [73] was used to estimate posterior probabilities. To avoid classification

bias due to the unbalanced data set (a minority 20% oddball class), a classification threshold for each

classifier was determined using the receiver operating characteristic (ROC) curve of the training data.

In particular, a threshold was determined by finding the point on the ROC curve which maximized

training sensitivity while maintaining training specificity above 60%.

Posterior probabilities of the test data were obtained for the 5 LDA and 5 SVM classifiers. For each

classifier, a test 15-trial average was classified into the oddball class if the posterior probability of the

test 15-trial average exceeded the classifier’s threshold, pre-determined with the ROC curve of the

training data. The 10 classifier decisions ym,m = 1, ..., 10, were combined with a weighted majority

vote for each test 15-trial average xn to arrive at an overall decision Yn,

Yn = sign

(10∑

m=1

wm ∗ ym(xn)

). (4.1)

Classifier weights wm were evaluated from classifier training errors for ntrain training 15-trial aver-

ages

αm =1

ntrain

∗∣∣∣{n|ym(xn) 6= tn}

∣∣∣ (4.2)

wm = 0.5 ∗ ln(

1− αm

αm

)(4.3)

where tn is the true label of each test 15-trial average and |.| denotes the cardinality of the set.

Equation 4.3 is valid for classifiers with low but above-chance accuracies (>50% for 2 classes) [?].

Offline classification results were obtained by averaging sensitivity, specificity and balanced accuracy

for 10 runs of 5-fold cross-validation. Sensitivity, specificity and balanced accuracy were defined

as

Sensitivity =TP

TP + FN(4.4)

41

Specificity =TN

TN + FP(4.5)

BalancedAccuracy =Sensitivity + Specificity

2(4.6)

where TP, FP, TN and FN are true positive, false positive, true negative and false negative, respec-

tively. A true positive indicated a correctly classified oddball 15-trial average.

4.3.5 Offline ERP Analysis

EEG data were re-referenced to both mastoids and band-pass filtered between 0.5 Hz and 30 Hz

using the EEGLAB Matlab toolbox [67]. In order to remove ocular artifacts from the EEG data,

the ADJUST algorithm was modified by restricting the spatial configuration to frontal electrodes

and by tuning artifact thresholds [66]. EEG data were also segmented into 500 ms stimulus-locked

trials and corrected with a 100 ms pre-stimulus baseline. Ten trials were then averaged per stimulus

condition.

In order to provide a standardized comparison of FOS and ERP classification results, the same

temporal and spectral features were extracted from EEG data. ERP features were extracted from 10-

trial averages of all 6 EEG channels. The ERP feature set contained a total of 270 features, including

24 spectral features (6 channels × 4 frequency bands) and 246 temporal features (6 channels × (1

time window × 9 features types + 4 time windows × 8 feature types)).

ERP feature normalization and selection were performed as described for the FOS and a SVM

classifier with a RBF kernel was used to classify the ERP data. The ERP classification posterior

probability threshold was determined with the training sensitivity and specificity as in the FOS case.

ERP balanced classification accuracy, sensitivity and specificity were obtained by performing 10 runs

of 5-fold cross-validation.

42

4.3.6 Online Feature Selection and Classification

Training data for online classification included data from a participant’s 3 offline sessions and the first

run of the respective online session, for a total of approximately 585 training 15-trial averages, with

20% oddball class 15-trial averages. Training features were normalized and the same feature normal-

ization parameters were applied in processing the test features during online classification.

Subsets of 10 features were selected using a FCBF for 100 runs of 5-fold cross-validation of the

training data. Features were ordered by frequency of selection within the 500 subsets of 10 features.

Final subsets for each of the 5 feature sets (4 measurements types in addition to their combination)

were obtained by keeping features which were selected more than 100 times. A minimum subset size

of 10 features was used as classification results were found to be lower with less than 10 features. If

less than 10 features were selected at least 100 times, the features with the next highest selection

count were picked. Subset features which were correlated with another higher ranked feature with a

Pearson’s correlation coefficient greater than 0.65 were removed from the subset.

Classifier training included evaluating the classifier parameters, posterior probability classification

thresholds and weights for the 5 SVM and 5 LDA classifiers. Online classifier testing was performed

immediately after each block using the classification procedure described for offline classification.

Each block consisted of 6 test 15-trial averages, including 1 oddball 15-trial average and 5 frequent 15-

trial averages. The first 15 oddball trials and the first 75 frequent trials within a block were averaged

in groups of 15 trials. The remaining trials in each block were not used for classification.

To provide classification feedback to the participant, a block was considered successfully classified if

the 1 oddball 15-trial average was correctly classified and if more than half of the 5 frequent 15-trial

averages were correctly classified. Online classification results were assessed in terms of balanced

accuracy, sensitivity and specificity per session (across 20 oddball 15-trial averages and 100 frequent

15-trial averages) as well as percentage of successfully classified blocks (across 20 blocks).

43

4.4 Results

4.4.1 ERP results

The ERP response to oddball images, reported in [72], was composed of a positive peak 200 ms

after stimulus onset and a negativity from 300-500 ms. ERP 10-trial averages from 2 sessions were

classified, differentiating 115 ± 5 oddball and 465 ± 17 frequent 10-trial averages per participant,

with an average balanced accuracy, sensitivity and specificity of 77 ± 5%, 82 ± 8% and 72 ± 6%,

respectively.

4.4.2 FOS Results

As reported in [72], FOS responses to oddball images contained a negativity between 150-250 ms.

FOS responses were classified offline with data from 3 sessions, providing an average of 113 ± 1

oddball and 468±4 frequent 15-trial averages per participant. Offline balanced accuracy was found to

remain constant or to increase for trial averages of up to 15 trials. The increase in balanced accuracy

was significant up to 20 averaged trials for 5 participants (Wilcoxon rank-sum test, p < 0.05).

Consequently, an average of 15 trials was selected for FOS offline and online classification as a

compromise between improving the SNR while maintaining a sufficient number of training data.

FOS 15-trial averages were classified offline, differentiating oddball and frequent image responses, with

an average balanced accuracy, sensitivity and specificity of 62±5%, 63±9% and 60±5%, respectively.

Offline weighted majority vote classification results per participant are shown in Figure 4.5 with the

online classification results. Offline FOS classification results for each participant were determined

to be significantly above chance with a permutation test (p < 0.05 for participant 12, p < 0.01 for all

other participants). Individual LDA sensitivities and specificities for each FOS measurement type are

shown in Figure 4.4. DC intensity feature sets yielded significantly higher LDA classification results

than did phase delay feature sets for 9 and 10 participants at 690 nm and 830 nm, respectively

(Wilcoxon rank-sum test, p < 0.05). LDA classification performance of 830 nm source feature sets

were significantly higher than that of 690 nm feature sets for 7 and 8 participants in DC intensity and

44

phase delay, respectively (Wilcoxon rank-sum test, p < 0.05). Classification performance differences

between measurement types were similar for LDA and SVM classifiers.

Figure 4.4: Individual classifier results per participant for each FOS feature set. (a) DC intensity at690 nm wavelength. (b) DC intensity at 830 nm wavelength. (c) Phase delay at 690 nm wavelength.(d) Phase delay at 830 nm wavelength. An asterisk (*) denotes a participant with DC intensityclassification results significantly higher than phase delay results. A λ denotes a participant with830 nm classification results significantly higher than 690 nm results.

FOS 15-trial averages of oddball versus frequent image responses were classified online during 2

sessions for 14 participants with an average balanced accuracy, sensitivity and specificity of 63± 6%,

64±15% and 63±6%, respectively. Online balanced accuracies were significantly better than chance

in both online sessions for 7 participants and in one online session for 5 participants (binomial test,

p < 0.05). Balanced accuracies for both online sessions are shown per participant in Figure 4.5.

Each online session was composed of 20 online visual oddball task blocks with classification feedback

provided to the participant following each block. Using a repeated measures ANOVA with a between-

group variable (participant), and a within-group variable (session), the participant effect was found

to be significant (F (1, 13) = 1724, p < 0.001, partial η2 = 0.99) but the session effect was not

significant (F (1, 13) = 3.005, p = 0.107, partial η2 = 0.19).

45

Figure 4.5: Online FOS balanced classification accuracy per session. The horizontal line denotes thechance level for the online sessions (binomial test, p < 0.05). Participant 3 did not attend the onlinesessions.

The dimensionality of selected features for online classification varied according to the selection

frequency of each feature during 100 runs of 5-fold cross-validation of a FCBF performed on training

data. Table 4.2 lists the selected features from the DC 690 nm and 830 nm features sets for each

participant during session 4, the first online session. The number of zero crossings (feature 8) and

the variance of the 0-500 ms window (feature 30) were selected for nearly all participants in both

DC 690 nm and DC 830 nm feature sets. The online selected feature dimensionality was consistently

between 10 and 19 features across all 5 feature sets and all participants.

Table 4.2: Feature types selected from DC 690 nm and 830 nm feature sets for online classificationduring the first online session (session 4). Superscripts denote the number of channels from which afeature type was selected. The number of zero crossings (feature 8) and the variance of the 0-500 mswindow (feature 30), both shown in bold font, were selected for nearly all participants in both DC690 nm and DC 830 nm features sets. Participant 3 did not attend the online sessions.

46

4.5 Discussion

This study determined the feasibility of automatically classifying the FOS during a visual oddball

task in order to differentiate responses to oddball and frequent images. ERPs were used to confirm

the presence of a prefrontal neuronal response to a visual oddball task that could be automatically

classified. An average ERP balanced classification accuracy of 77 ± 5% for 10-trial averages was

obtained across all participants. The obtained accuracy was lower than typical ERP BCI accuracies,

which range from 80% for single-trial classification to 100% for 10-trial average classification [52,

74], since in this study ERP measurements were restricted to 6 channels in the prefrontal cortex.

Nonetheless, these results demonstrated the feasibility of classifying a prefrontal ERP response with

an accuracy above 70%, which is the required accuracy for effective BCI communication [75].

This study is one of the first reported attempts to automatic classify the FOS. FOS responses to odd-

ball and frequent images were classified offline with an average balanced accuracy of 62±5%. Offline

balanced accuracies were significantly above chance for all participants (permutation test, p < 0.05

for 1 and p < 0.01 for 14 participants), which confirms that automatic detection of the FOS is possible

at above-chance levels. The ability to automatically detect and classify the FOS also confirms the

presence of a FOS response which corresponds with ERPs. However, FOS offline balanced accuracies

did not reach the 70% accuracy threshold required for effective BCI communication [75].

Although 15 trials were averaged for classification, it may be possible to reduce the multiplicity of

trials as increasing the number of averaged trials only improved classification results for 5 of 15 partic-

ipants. Increasing the number of averaged trials above 15 did not improve accuracies as this reduced

the number of available training data for classification. Averaging 15 trials to detect a FOS response

represents a significant reduction in trial averaging compared to previous FOS studies, which ranged

between 100 [37] and 1000 trials [41] for FOS detection. Furthermore, mainly grand average FOS

responses across participants are reported in the FOS literature. Automatic classification therefore

enables reporting FOS responses on a per participant and per session basis as significantly fewer

averaged trials are required for detection.

47

Both DC intensity and phase delay measurements were classified even though a FOS response was

not found with phase delay in a previous analysis of this data [72]. As expected, individual classifier

results were significantly higher for DC intensity than phase delay for 9 participants. Phase delay

classifiers were included in the weighted majority vote even though they had lower individual classifier

results, as they either improved or maintained the weighted majority vote classification results for all

participants. The superior DC intensity results contradict the findings of early FOS studies which

have suggested that the FOS is best measured with phase delay [20, 25, 29]. However, Gratton et

al. attributed their lack of FOS DC intensity findings to external sources of noise, such as external

light [25]. In the present study, precautions were taken to remove external sources of light, including

collecting data in a dark room and placing a black cloth over the NIRS headband. Consistent with

the present results, other FOS studies have reported FOS detection with DC intensity but not phase

delay [30, 58]. Using Monte Carlo simulations, Franceschini et al. demonstrated that scattering

changes induced smaller changes in phase delay than intensity, thus explaining the lower SNR of

phase delay compared to intensity [43].

Classifiers of 830 nm wavelength feature sets also had significantly higher classification results than

690 nm feature sets for half of the participants. This finding is in agreement with previous literature

suggesting that FOS detection is improved at higher wavelengths [17, 29]. It was proposed that

FOS sensitivity is higher at 830 nm wavelengths compared to 690 nm as there is increased spectra

absorption at 690 nm which leads to fewer photons reaching the detector and thus a lower SNR [26].

The wavelength-dependent classification results also eliminated the possibility that the classified

responses were due to motion artifacts as the paired-wavelength sources would have been affected in

like manner by motion artifacts.

An average online balanced accuracy of 63± 6% was achieved for this first reported attempt to clas-

sify the FOS online. Online FOS balanced classification accuracies were significantly above chance

in both online sessions for 7 participants and in one online session for 3 participants (binomial test,

p < 0.05). Poor SNR as well as unexpected feedback (due to incorrect classification) may have

contributed to unsuccessful online FOS classification for some participants. The low percentages

of successfully classified blocks led to increased frustration for some participants, which may have

48

affected classification results as both fatigue and frustration have been reported to affect BCI perfor-

mance [76]. It also may not be possible to reliably classify the FOS response for some participants

due to low SNR.

Online and offline FOS participant-specific classification results were similar across different sessions

indicating a consistent FOS response across offline and online sessions, which in turn supports the

use of data from multiple sessions for classifier training. There was no significant difference in

classification results between online sessions of each participant. However, there was a significant

difference in online classification results between participants. These findings are consistent with

the ERP literature, as ERPs have also been shown to be reproducible across sessions, with greater

consistency for amplitude than latency [77]. Additionally, participant-specific data are necessary for

ERP classification due to larger inter-participants variations [78]. ERP latency variations between

participants have been found to relate to stimulus reaction time [79, 80]. Therefore, FOS between-

participant variations may relate to participant-specific mental states and performance of the visual

oddball task.

The number of zero crossings and the variance from 0-500 ms features were selected online across

multiple channels in both DC 830 nm and DC 690 nm feature sets for nearly all participants. The

number of zero crossings was found to be lower and the variance higher for oddballs compared to

frequent FOS responses. The number of zero crossings corresponds to the low-frequency correlations

found between the FOS and ERP oddball responses in [72]. Similarly, event-related oscillations have

exhibited power increases in upper delta and theta frequency band power in response to oddball

target stimuli in the frontal cortex [50, 51]. The increase in variance may correspond to an overall

increase in signal energy in response to oddball stimuli while the number of zero crossing decrease

may correspond to an upper delta and theta spectral increase relative to other frequency bands.

Although above-chance online balanced accuracies where achieved for some participants, the low SNR

of the FOS poses significant limitations for automatic classification in terms of information transfer

rate, as trial averaging is required. Additionally, averaging may limit the type of task paradigm that

can be used in future FOS BCIs. As larger ERPs have been found in fronto-central and parietal

areas of the brain and higher classification results have been obtained with these ERPs, future FOS

49

classification studies should include additional channels over the fronto-central and parietal areas.

A larger FOS response may be found in these areas, thus improving the SNR while potentially

reducing the number of averaged trials. Another limitation of this study was the imbalanced task

paradigm, necessitated using a posterior probability threshold to avoid classification bias towards the

frequent class. Avoiding an imbalanced task paradigm could potentially improve results in future

FOS classification studies and reduce the required number of training data. Another approach to

reduce the required training data per participant while increasing the number of training data would

be a generic classification model across participants, where a classifier would be trained with data

from all participants.

4.6 Conclusion

In this study, a framework for automatically classifying the FOS was evaluated. Prefrontal FOS

responses to a visual oddball task were first classified offline with data from 3 sessions. Offline

session data were then used to train classifiers for the classification of FOS responses to oddball

versus frequent images in 2 online sessions where classification feedback was provided to participants.

An average online balanced classification accuracy of 63 ± 6% was achieved across all participants,

which did not meet the 70% accuracy level required for effective BCI communication. Although

the FOS has high temporal and spatial resolution, accuracy and information transfer rate of FOS

BCIs remain limited due to the low SNR. Future FOS-BCI studies should include additional FOS

measurement channels across the fronto-central and parietal cortices, using a balanced task paradigm

and further investigate spectral features.

50

Chapter 5

Conclusion

5.1 Contributions

This thesis was focused on detection and automatic classification of the FOS measured with NIRS.

The main contributions of this thesis were as follows:

1. Evaluated the temporal and spectral relationships of FOS-ERP prefrontal responses to a visual

oddball task. To our knowledge, this is the first investigation of spectral FOS-ERP relationships

in the prefrontal cortex, which were found to be more consistent than temporal correlations

across participants.

2. Developed and evaluated an algorithm for automatic FOS classification, demonstrating that

automatic classification of the FOS can be achieved at above-chance levels. To our knowledge,

this is the first reporting of FOS automatic classification. This work provides a framework for

future FOS classification studies.

3. Evaluated FOS classification in an online BCI context, determining that online FOS classifica-

tion could be achieved at above-chance levels but not for all participants. To our knowledge,

this is the first reporting of online classification of the NIRS FOS.

51

5.2 Future Work

The following future directions for FOS classification research are proposed based on the findings of

this thesis.

5.2.1 Further investigation of FOS spectral features

The first part of this thesis investigated FOS-ERP oddball correlations in the temporal and spectral

domains. Low-frequency FOS-ERP correlations of oddball responses were found in the upper delta

and theta frequency bands. Spectral FOS-ERP correlations were more consistent across participants

than the evaluated temporal correlations. Additionally, through evaluating the FOS classification

algorithm, a spectral feature, the number of zero crossings, was selected from both 690 nm and 830 nm

feature sets across nearly all participants. Oddball responses were found to have a lower number of

zero crossings than frequent responses which was congruous with the significant low frequency FOS-

ERP correlations for oddball responses. As spectral relationships and a spectral feature were found

to be more consistent across participants than their temporal counterpart, exploring various types of

spectral features may lead to enhanced FOS classification accuracies.

5.2.2 Measurement of additional cortical locations

In this thesis, FOS and ERP prefrontal responses to oddball stimuli were detected and classified

with accuracies at above-chance levels but these accuracies did not achieve levels of effective BCI

communication. Although FOS and ERPs have been previously reported in the prefrontal cortex

[35,48,49], most ERP studies focus on the fronto-central and parietal regions. Larger ERP responses

have been reported in the fronto-central and parietal regions, which has contributed to higher ERP

classification accuracies [52, 74]. Hence, future FOS classification studies should measure the FOS

across additional measurement channels over the fronto-central and parietal regions of the brain.

This may facilitate improvements in SNR, and in turn, a reduction in the multiplicity of trials.

52

5.2.3 Potential FOS-BCI paradigms

This thesis demonstrated that automatic FOS classification could be achieved online at above-chance

accuracy levels. However, FOS classification accuracies were limited by the low SNR of the FOS,

thus efficient BCI communication accuracy levels (>70%) were not reached. If the FOS SNR could

be improved, the high temporal and spatial resolution of the FOS would provide the opportunity

for a multi-class BCI paradigm with a high information transfer rate. BCIs combining the FOS and

the hemodynamic signal would also be worth exploring. A potential task paradigm for these BCIs

would be the Stroop task as both signals have been detected with this paradigm. The FOS could

also be used for error detection as proposed in [33], while using the hemodynamic signal as the main

BCI control signal. As both signals are measured with NIRS, these proposed BCI paradigms would

be more realistic for the clinical setting than multi-modal BCIs, which require 2 or more types of

instrumentation.

5.2.4 FOS studies with the target population

In this thesis, the FOS studies were completed with able-bodied adult subjects. All FOS studies

found in the literature have also been completed with able-bodied adult subjects, thus the FOS

has yet to be studied with individuals who have severe motor impairment. As individuals with

severe motor impairment are the intended user population for brain-computer interfaces, it would

be important to ensure that the FOS can be detected with these individuals before further pursuing

FOS-BCIs. The SNR of the FOS may be even lower for individuals with motor impairments due

to motion artifacts from involuntary or spastic movements. There may also be differences in the

characteristics of the FOS response, such as the latency, and therefore FOS-BCIs would need to be

adapted accordingly.

53

Bibliography

[1] U. Chaudhary, N. Birbaumer, and M.R. Curado. Brain-machine interface (BMI) in paralysis.

Annals of Physical Rehabilitation Medicine, 58:9–13, 2015.

[2] Eric W. Sellers, David B. Ryan, and Christopher K. Hauser. Noninvasive brain-computer inter-

face enables communication after brainstem stroke. Neurotechnology, 6(257):7pp, Oct 2014.

[3] Jackie Parkes, Nan Hill, Mary Jane Platt, and Caroline Donnelly. Oromotor dysfunction and

communication impairments in children with cerebral palsy: a register study. Developmental

Medicine & Child Neurology, pages 1113–1119, 2010.

[4] S. Moghimi, A. Kushki, A. M. Guerguerian, and T. Chau. A review of EEG-based brain-

computer interfaces as access pathways for individuals with severe disabilities. Assistive Tech-

nology, 25(2):99–110, May 2013.

[5] Luis Fernando, Nicolas Alonso, and Jaime Gomez-Gil. Brain computer interfaces, a review.

Sensors, pages 1211–1279, 2012.

[6] Shirley Coyle, Tomas Ward, Charles Markham, and Gary McDarby. On the suitability of

near-infrared (NIR) systems for next-generation brain-computer interfaces. Physiological Mea-

surement, 25:815–822, Jul 2004.

[7] Gabriele Gratton and Monica Fabiani. Fast optical imaging of human brain function. Frontiers

in Human Neuroscience, 4:52, 2010.

[8] Gabriele Gratton, Emily Wee, Elena I. Rykhlevskaia, Echo E. Leaver, and Monica Fabiani. Does

white matter matter? spatio-temporal dynamics of task switching in aging. Journal of Cognitive

54

Neuroscience, 21(7):1380–1395, Jul 2009.

[9] Simone Cutini and Sabrina Brigadoi. Unleashing the future potential of functional near-infrared

spectroscopy in brain sciences. Journal of Neuroscience Methods, 232:152–156, 2014.

[10] Noman Naseer and Keum-Shik Hong. fNIRS-based brain-computer interfaces: a review. Fron-

tiers in Human Neuroscience, 9:15pp, 2015.

[11] Edward L. Maclin, Gabriele Gratton, and Monica Fabiani. Optimum filtering for EROS mea-

surements. Psychophysiology, 40, 542-547 2003.

[12] Megan Strait and Matthias Scheutz. What we can and cannot (yet) do with functional near

infrared spectroscopy. Frontiers in Human Neuroscience, 8(12pp), 2014.

[13] Gunther Bauernfeind, Robert Leeb, Selina Christin Wriessnegger, and Gert Pfurtscheller. De-

velopment, set-up and first results for a one-channel near-infrared spectroscopy system. Biomed

Tech, 53:36–43, 2008.

[14] Nima Hemmati Berivanlou, Seyed Kamaledin Setarehdana, and Hossein Ahmadi Noubari.

Evoked hemodynamic response estimation using ensemble empirical mode decomposition based

adaptive algorithm applied to dual channel functional near infrared spectroscopy (fNIRS). Jour-

nal of Neuroscience Methods, 224:13–25, 2014.

[15] ISS Inc. Imagent: Functional brain imaging system using infrared photons, 2015.

[16] L.B. Cohen. Changes in neuron structure during action potential propagation and synaptic

transmission. Physiological Reviews, 53(2):373–418, 1973.

[17] Jonghwan Lee and Sung June Kim. Spectrum measurement of fast optical signal of neural

activity in brain tissue and its theoretical origin. NeuroImage, 51:713–722, 2010.

[18] Xiao-Su Hu, Keum-Shik Hong, and Shuzhi Sam Ge. Recognition of stimulus-evoked neuronal op-

tical response by identifying chaos levels of near-infrared spectroscopy time series. Neuroscience

Letters, 504(2):115–120, Oct 2011.

55

[19] Bailei Sun, Lei Zhang, Hui Gong, Jinyan Sun, and Qingming Luo. Detection of optical neuronal

signals in the visual cortex using continuous wave near-infrared spectroscopy. NeuroImage,

87(0):190–198, 2014.

[20] G. Gratton, M. Fabiani, D. Friedman, M. A. Franceschini, S. Fantini, P. Corballis, and E. Grat-

ton. Rapid changes of optical parameters in the human brain during a tapping task. Journal of

Cognitive Neuroscience, 7(4):446–456, 1995.

[21] G. Gratton, Paul M. Corballis, Eunhee Cho, Monica Fabiani, and Donald C. Hood. Shades of

gray matter: Noninvasive optical images of human brain responses during visual stimulation.

Psychophysiology, 32:505–509, 1995.

[22] Antonio Maria Chiarelli, Assunta Di Vacri, Gian Luca Romani, and Arcangelo Merla. Fast

optical signal in visual cortex: Improving detection by general linear convolution model. Neu-

roImage, 66:194–202, 2013.

[23] Gabriele Gratton and Paul M. Corballis. Removing the heart from the brain: Compensation for

the pulse artifact in the photon migration signal. Psychophysiology, 32:292–299, 1995.

[24] Gabriele Gratton and Monica Fabiani. The event-related optical signal (EROS) in visual cortex:

replicability, consistency, localization, and resolution. Psychophysiology, 40(4):561–571, Jul 2003.

[25] Gabriele Gratton, Carrie R. Brumback, Brian A. Gordon, Melanie A. Pearson, Kathy A. Low,

and Monica Fabiani. Effects of measurement method, wavelength, and source-detector distance

on the fast optical signal. NeuroImage, 32(4):1576–1590, Oct 2006.

[26] Edward L. Maclin, Kathy A. Low, Monica Fabiani, and Gabriele Gratton. Improving the signal-

to-noise ratio of event-related optical signals. IEEE Engineering in Medecine and Biology Mag-

azine, 26(4):47–51, 2007.

[27] Elena Rykhlevskaia, Monica Fabiani, and Gabriele Gratton. Lagged covariance structure models

for studhying functional connectivity in the brain. NeuroImage, 30:1203–1218, Jan 2006.

[28] Chun-Yu Tse, Brian A. Gordon, Monica Fabiani, and Gabriele Gratton. Frequency analysis

of the visual steady-state response measured with the fast optical signal in younger and older

56

adults. Biological Psychology, 85:79–89, 2010.

[29] Antonio Maria Chiarelli, Gian Luca Romani, and Arcangelo Merla. Fast optical signals in

the sensorimotor cortex: General linear convolution model applied to multiple source-detector

distance-based data. NeuroImage, 85 Pt 1:245–254, Jan 2014.

[30] G. Morren, M. Wolf, P. Lemmerling, U. Wolf, J.H. Choi, and G. Gratton. Detection of fast neu-

ronal signals in the motor cortex from functional near infrared spectroscopy measurements using

independent componentanalysis. Medical and Biological Engineering and Computing, 42:92–99,

2004.

[31] Nathan A. Parks, Edward L. Maclin, Kathy A. Low, Diane M. Beck, Monica Fabiani, and

Gabriele Gratton. Examining cortical dynamics and connectivity with simultaneous single-pulse

transcranial magnetic stimulation and fast optical imaging. NeuroImage, 59:2504–2510, 2012.

[32] Zhongxing Zhang, Bailei Sun, Hui Gong, Lei Zhang, Jinyan Sun, Bangde Wang, and Qingming

Luo. A fast neuronal signal-sensitive continuous-wave near-infrared imaging system. Review of

Scientific Instruments, 83:8pp, 2012.

[33] Pauline L Baniqued, Kathy A Low, Monica Fabiani, and Gabriele Gratton. Frontoparietal

traffic signals: a fast optical imaging study of preparatory dynamics in response mode switching.

Journal of Cognitive Neuroscience, 25(6):887–902, Jun 2013.

[34] Jian Huang, Suiping Wang, Shiwei Jia, Deyuan mo, and Hsuan-Chih Chen. Cortical dynamics of

semantic processing during sentence comprehension: Evidence from event-related optical signals.

PLoS One, 8(8):10pp, Aug 2013.

[35] Kathy A Low, Echo Leaver, Arthur F Kramer, Monica Fabiani, and Gabriele Gratton. Fast

optical imaging of frontal cortex during active and passive oddball tasks. Psychophysiology,

43(2):127–136, Mar 2006.

[36] Andrei V. Medvedev, Jana Kainerstorfer, Sergey V. Borisov, Randall L. Barbour, and John Van-

Meter. Event-related fast optical signal in a rapid object recognition task: Improving detection

by the independent component analysis. Brain Research, 1236:145–158, 2008.

57

[37] Andrei V Medvedev, Jana M Kainerstorfer, Sergey V Borisov, Amir H Gandjbakhche, and John

Vanmeter. ”seeing” electroencephalogram through the skull: imaging prefrontal cortex with fast

optical signal. Journal of Biomedical Optics, 15(6):061702–1 – 061702–12, 2010.

[38] Chun-Yu Tse, Kei-Rui Tien, and Trevor B Penney. Event-related optical imaging reveals the

temporal dynamics of right temporal and frontal cortex activation in pre-attentive change de-

tection. NeuroImage, 29(1):314–320, Jan 2006.

[39] Chun-Yu Tse and Trevor B Penney. Preattentive change detection using the event-related optical

signal. IEEE Engineering in Medecine and Biology Magazine, pages 52–58, 2007.

[40] Chun-Yu Tse, Teemu Rinne, Kwun Kei Ng, and Trevor B Penney. The functional role of the

frontal cortex in pre-attentive auditory change detection. NeuroImage, 83:870–879, Dec 2013.

[41] Jens Steinbrink, Florian C D. Kempf, Arno Villringer, and Hellmuth Obrig. The fast opti-

cal signal–robust or elusive when non-invasively measured in the human adult? NeuroImage,

26(4):996–1008, Jul 2005.

[42] Harsha Radhakrishnan, Wim Vanduffel, Hong Ping Deng, Leeland Ekstrom, David A. Boas,

and Maria Angela Franceschini. Fast optical signal not detected in awake behaving monkeys.

NeuroImage, 45:410–419, 2009.

[43] Maria Angela Franceschini and David A. Boas. Noninvasive measurement of neuronal activity

with near-infrared optical imaging. NeuroImage, 21:372–386, 2004.

[44] Andrea Malagoli. Studies for a low-cost instrumental setup for recording and processing the fast

optical response of cortical neurons in visual stimulation. PhD thesis, Universita Degli Studi Di

Modena E Reggio Emilia, 2009.

[45] Marcel van Gerven, Jason Farquhar, Rebecca Schaefer, Rutger Vlek, Jeroen Geuze, Anton

Nijholt, Nick Ramsey, Pim Haselager, Louis Vuurpijl, Stan Gielen, and Peter Desain. The

brain-computer interface cycle. Journal of Neural Engineering, 6:10pp, 2009.

[46] Scott A. Huettel and Gregory McCarthy. What is odd in the oddball task? prefrontal cortex is

activated by dynamic changes in response strategy. Neuropsychologia, 42:379–386, 2004.

58

[47] Eiji Kirino, Aysenil Belger, Patricia Goldman-Rakic, and Gregory McCarthy. Prefrontal activa-

tion evoked by infrequent target and novel stimuli in a visual target detection task: An event-

related functional magnetic resonance imaging study. The Journal of Neuroscience, 20(17):6612–

6618, 2000.

[48] Geoffrey F. Potts. An ERP index of task relevance evaluation of visual stimuli. Brain and

Cognition, 56:5–13, 2004.

[49] Maurizio Codispoti, Vera Ferrari, Markus Junghofer, and Harald T. Schupp. The categorization

of natural scenes: Brain attention networks revealed by dense sensor ERPs. NeuroImage, 32(583-

591), 2006.

[50] Kevin A. Jones, Bernice Porjesz, David Chorlian, Madhavi Rangaswamy, Chella Kamarajan,

Ajayan Padmanabhapillai, Arthur Stimus, and Henri Begleiter. S-transform time-frequency

analysis of P300 reveals deficits in individuals diagnosed with alcoholism. Clinical Neurophysi-

ology, 117:2128–2143, 2006.

[51] Edward M. Bernat, Stephen M. Malone, William J. Williams, Christopher J. Patrick, and

William G. Iacono. Decomposing delta, theta, and alpha time-frequency ERP activity from

a visual oddball task using PCA. International Journal of Psychophysiology, 64:62–74, 2007.

[52] T Kaufmann, S M Schulz, C Grunzinger, and A Kubler. Flashing characters with famous faces

improves ERP-based brain-computer interface performance. Journal of Neural Engineering,

8:10pp, 2011.

[53] L. B. Cohen, R. D. Keynes, and D. Landowne. Changes in light scattering that accompany the

action potential in squid giant axons: potential-dependent ccomponents. Journal of Physiology,

224:701–725, 1972.

[54] R. A. Stepnoski, A. Laporta, F. Raccuia-Behling, G. E. Blonder, R. E. Slusher, and D. Kleinfeld.

Noninvasive detection of changes in membrane potential in cultured neurons by light scattering.

Proc. Natl. Acad. Sci., 88(21):9382–9386, 1991.

[55] David M/ Rector, Gina R. Poe, Morten P. Kristensen, and Ronald M. Harper. Light scattering

59

changes follow evoked potentials from hippocampal schaeffer collateral stimulation. Journal of

Neurophysiology, 78(3):1707–1713, 1997.

[56] Edward L. Maclin, Kathy A. Low, Jeffrey J. Sable, Monica Fabiani, and Gabriele Gratton. The

event-related optical signal to electrical stimulation of the median nerve. NeuroImage, 21:1798–

1894, 2004.

[57] M. Kohla J. Steinbrink and, H. Obriga, G. Curioc, F. Syre, F. Thomasa, H. Wabnitzb, H. Rin-

nebergb, and A. Villringera. Somatosensory evoked fast optical intensity changes detected non-

invasively in the adult human head. Neuroscience Letters, 291:105–108, 2000.

[58] Martin Wolf, Ursula Wolf, Jee H. Choi, Vladislav Toronov, L. Adelina Paunesco, Antonios

Michalos, and Enrico Gratton. Fast cerebral functional signal in the 100-ms range detected

in the visual cortex by frequency-domain nearinfrared spectrophotometry. Psychophysiology,

40:521–528, 2003.

[59] Gabriele Gratton, Anita Sarno, Ed Maclin, Paul M. Corballis, and Monica Fabiani. Toward

noninvasive 3-D imaging of the time course of cortical activity: Investigation of the depth of the

event-related optical signal. NeuroImage, 11(5):491–504, 2000.

[60] Kathy A Low, Echo Leaver, Arthur F Kramer, Monica Fabiani, and Gabriele Gratton. Share

or compete? load-dependent recruitment of prefrontal cortex durign dual-task performance.

Psychophysiology, 46:1069–1079, 2009.

[61] G. Gratton and M. Fabiani. Shedding light on brain function: the event-related optical signal.

Trends in Cognitive Science, 5(8):357–363, Aug 2001.

[62] Monica Fabiani, Brian A. Gordon, Edward L. Maclin, Melanie A. Pearson, Carrie R. Brumback-

Peltz, Kathy A. Low, Edward McAuley, Bradley P. Sutton, Arthur F. Kramer, and Gabriele

Gratton. Neurovascular coupling in normal aging: A combined optical, ERP and fMRI study.

NeuroImage, 85:592–607, 2014.

[63] Chun-Yu Tse and Trevor B. Penney. On the functional role of temporal and frontal cortex

activation in passive detection of auditory deviance. NeuroImage, 41:1462–1470, 2008.

60

[64] Independent component analysis (ICA) and blind source separation (BSS), 3 2013.

[65] Hojjat Adeli, Ziqin Zhou, and Nahid Dadmehr. Analysis of EEG records in an epileptic patient

using wavelet transform. Journal of Neuroscience Methods, 123:69–87, 2003.

[66] Andrea Mognon, Jorge Jovicich, Lorenzo Bruzzone, and Marco Buiatti. ADJUST: An automatic

EEG artifact detector based on the joint use of spatial and temporal features. Psychophysiology,

48:229–240, 2011.

[67] A Delorme and S Makeig. EEGLAB: an open source toolbox for analysis of single-trial EEG

dynamics. Journal of Neuroscience Methods, 134:9–21, 2004.

[68] R. Oostenveld, P. Fries, E. Maris, and JM Schoffelen. FieldTrip: Open source software for

advanced analysis of MEG, EEG, and invasive electrophysiological data. Computational Intelli-

gence and Neuroscience, 2011, 2011.

[69] Conrado A. Bosman, Jan-Mathijs Schoffelen, Nicolas Brunet, Robert Oostenveld, Andre M.

Bastos, Thilo Womelsdorf, Birthe Rubehn, Thomas Stieglitz, Peter De Weerd, and Pascal Fries.

Attentional stimulus selection through selective synchronization between monkey visual areas.

Neuron, 75:875–888, 2012.

[70] Geoffrey F. Potts, Mario Lioiti, Don M. Tucker, and Michael I. Posner. Frontal and inferior

temporal cortical activity in visual target detection: Evidence from high spatially sampled event-

related potentials. Brain Topography, 9(1):3–14, 1996.

[71] John Polich. Updating P300: An integrative theory of P3a and P3b. Clinical Neurophysiology,

118:2128–2148, 2007.

[72] Proulx N. and Chau T. Quantifying fast optical signal and event-related potential correlations

durign a visual oddball task. Submitted for publication.

[73] Ting-Fan Wu, Chih-Jen Lin, and Ruby C. Weng. Probability estimates for multi-class classifi-

cation by pairwise coupling. J. Mach. Learn. Res., 5:975–1005, December 2004.

[74] Lynn M. McCane, Susan M. Heckman, Dennis J. McFarland, George Townsend, Joseph N. Mak,

Eric W. Sellers, Debra Zeitlin, Laura M. Tenteromano, Jonathan R. Wolpaw, and Theresa M.

61

Vaughan. P300-based brain-computer interface (BCI) event-related potentials (ERPs): People

with amyotrophic lateral sclerosis (ALS) vs. age-matched controls. Clinical Neurophysiology,

126(2124-2131), 2015.

[75] A. Kubler, V. K. Mushahwar, L.R. Hochberg, and J. P. Donoghue. BCI meeting 2005 - workshop

on clinical issues and applications. IEEE Transactions on Neural Systems and Rehabilitation

Engineering, 14(2):131–134, 2006.

[76] Andrew Myrden and Tom Chau. Effects of user mental state on EEG-BCI performance. Frontiers

in Human Neuroscience, 9(308):11pp, 2015.

[77] L. M. WILLIAMS, E. SIMMS, C. R. CLARK, R. H. PAUL, D. ROWE, and E. GORDON.

The test-retest reliability of a standardized neurocognitive and neurophysiological test battery:

“neuromarker”. International Journal of Neuroscience, 115:1605–1630, 2005.

[78] Hubert Cecotti, Amar R. Marathe, and Anthony J. Ries. Optimization of single-trial detection

of event-related potentials through artificial trials. IEEE TRANSACTIONS ON BIOMEDICAL

ENGINEERING, 62(9):2170–2176, 2015.

[79] Scott Makeig, Marissa Westerfield, Tzyy-Ping Jung, James Covington, Jeanne Townsend, Ter-

rence J. Sejnowski, and Eric Courchesne. Functionally independent components of the late

positive event-related potential during visual spatial attention. The Journal of Neuroscience,

19(7):2665–2680, 1999.

[80] Tzyy-Ping Jung, Scott Makeig, Marissa Westerfield, Jeanne Townsend, Eric Courchesne, , and

Terrence J. Sejnowski. Analysis and visualization of single-trial event-related potentials. Human

Brain Mapping, 14:166–185, 2001.

62