Enhanced Performance of a Brain Switch by Simultaneous Use of …ineuro.korea.ac.kr/theme/community/Journal paper/47.pdf · 2020. 10. 12. · such as ﬂickering visual stimuli or

2102 IEEE TRANSACTIONS ON NEURAL SYSTEMS AND REHABILITATION ENGINEERING, VOL. 28, NO. 10, OCTOBER 2020

Enhanced Performance of a Brain Switch bySimultaneous Use of EEG and NIRS Data for

Asynchronous Brain-Computer InterfaceChang-Hee Han , Klaus-Robert Müller, and Han-Jeong Hwang

Abstract— Previous studies have shown the superiorperformance of hybrid electroencephalography (EEG)/near-infrared spectroscopy (NIRS) brain-computer inter-faces (BCIs). However, it has been veiled whether the use ofa hybrid EEG/NIRS modality can provide better performancefor a brain switch that can detect the onset of the intentionto turn on a BCI. In this study, we developed such a hybridEEG/NIRS brain switch and compared its performance withsingle modality EEG- and NIRS-based brain switch respec-tively, in terms of true positive rate (TPR), false positiverate (FPR), onset detection time (ODT), and informationtransfer rate (ITR). In an offline analysis, the performance ofa hybrid EEG/NIRS brain switch was significantly improvedover that of EEG- and NIRS-based brain switches in general,and in particular a significantly lower FPR was observedfor the hybrid EEG/NIRS brain switch. A pseudo-onlineanalysis was additionally performed to confirm the feasi-bility of implementing an online BCI system with our hybridEEG/NIRS brain switch. The overall trend of pseudo-onlineanalysis results generally coincided with that of the offlineanalysis results. No significant difference in all performancemeasures was also found between offline and pseudo onlineanalysis schemes when the amount of training data wassame, with one exception for the ITRs of an EEG brainswitch. These offline and pseudo-online results demon-strate that a hybrid EEG/NIRS brain switch can be used toprovide a better onset detection performance than that of asingle neuroimaging modality.

Index Terms— Brain-computer interface (BCI), electroen-cephalography (EEG), near-infrared spectroscopy (NIRS),brain switch, onset detection.

Manuscript received October 25, 2019; revised April 1, 2020 andJune 26, 2020; accepted August 10, 2020. Date of publication August 17,2020; date of current version October 8, 2020. This work was supportedby the Institute for Information & Communications Technology Plan-ning & Evaluation (IITP) grant funded by the Korea government(No. 2017-0-00451; Development of BCI based Brain and Cogni-tive Computing Technology for Recognizing User’s Intentions usingDeep Learning), and by the Basic Research Program through theNational Research Foundation of Korea (NRF) funded by the MSIT(2020R1A4A1017775). (Corresponding authors: Klaus-Robert Müller;Han-Jeong Hwang.)

Chang-Hee Han is with the Machine Learning Group, TechnischeUniversität Berlin, 10587 Berlin, Germany.

Klaus-Robert Müller is with the Machine Learning Group, TechnischeUniversität Berlin, 10587 Berlin, Germany, also with the Department ofBrain and Cognitive Engineering, Korea University, Seoul 136-713, SouthKorea, and also with the Max Planck Institute for Informatics, 66123 Saar-brücken, Germany (e-mail: [email protected]).

Han-Jeong Hwang is with the Department of Electronics and Informa-tion Engineering, Korea University, Sejong 30019, South Korea (e-mail:[email protected]).

Digital Object Identifier 10.1109/TNSRE.2020.3017167

I. INTRODUCTION

BRAIN-COMPUTER interfaces (BCIs) are a state-of-the-art technology that interprets human intention using brainsignals. BCI can provide alternative communication channelsfor patients in locked-in syndrome (LIS) caused by neurode-generative diseases, such as amyotrophic lateral sclerosis(ALS) because BCI does not need voluntary body movementsto control external devices for communication purposes[2]–[4]. For the past decades, promising BCI paradigms andapplications have been developed using various neuroimagingmodalities, such as electroencephalography (EEG)(e.g. [5]–[8]), magnetoencephalography (MEG) (e.g. [9], [10]),functional near-infrared spectroscopy (fNIRS) (e.g. [11]–[14]),and functional magnetic resonance imaging (fMRI)(e.g. [15]–[17]).

BCIs can be developed under one of two differentoperation paradigms: i) synchronous or ii) asynchronous para-digm [18]–[23]. A synchronous BCI system evaluates the con-trol intentions of a user only within predefined time periods,meaning that users cannot control a BCI system wheneverthey want. On the other hand, asynchronous BCI systemscontinuously monitor a user’s control intentions during theentire period, and thus users can freely operate BCI systemswithout time constraints whenever they want. For this rea-son, asynchronous paradigms have been considered as morerealistic and practical [20].

However, unfortunately, the performance of asynchronousBCI systems is generally worse than that of synchronousBCI systems, showing a higher false positive rate (FPR) andlower true positive rate (TPR) [24], [25]. Especially, a highFPR is a critical problem when using asynchronous BCIsystems, since false operations that are not matched with theuser’s intention are likely to occur. Once FPs are produced,an additional BCI command is required to correct it [26], [27],which is time-consuming. Furthermore, the FPR can signifi-cantly increase e.g. when a multi-command BCI system isused because a BCI system has to monitor all commandssimultaneously.

To develop an asynchronous BCI system with a low FPR,researchers have focused on the development of a two-stepapproach by introducing a brain switch in an asynchronousBCI system [28]–[35]. A two-step approach consists of anidle state and a control state. In an idle state, a brain switch

1534-4320 © 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.

Authorized licensed use limited to: Korea University. Downloaded on October 09,2020 at 09:46:28 UTC from IEEE Xplore. Restrictions apply.

https://orcid.org/0000-0001-8668-3989https://orcid.org/0000-0002-1183-1219

HAN et al.: ENHANCED PERFORMANCE OF A BRAIN SWITCH BY SIMULTANEOUS USE OF EEG AND NIRS DATA FOR ASYNCHRONOUS BCI 2103

embedded in an asynchronous BCI system detects an onsetintention that switches an asynchronous BCI system intooperational mode. During periods of idle state detection, theasynchronous BCI system cannot be operated and thereby falsepositive detections are prevented. Once the onset intention isdetected by a brain switch, an idle state is converted intoa control state, and an asynchronous BCI system starts toidentify user’s control intentions that will then operate an asyn-chronous BCI system. Otherwise, an idle state is continuouslyretained and the FPRs of asynchronous BCI systems can besignificantly reduced [29].

Brain switches that are the core of this two-step approachhave been developed using various types of brain activ-ity, such as event-related potential (ERP) [32], steady-statevisual evoked potential (SSVEP) [31], [36], [37], sensorimotorrhythm (SMR) [30], [38]–[40], and change in hemodynamicresponse [41]–[43]. Brain switches can be developed basedon two different paradigms according to whether externalstimuli are used or not. SSVEP and ERP have been used todevelop exogenous brain switches that use external stimuli,such as flickering visual stimuli or auditory beeps to evokediscriminative brain patterns, and they have shown reliableperformance [31], [32], [36], [37]. However, exogenous brainswitches can cause user fatigue or headache due to the repet-itive presentation of external stimuli [44]–[46]. In addition,exogenous brain switches cannot be fully utilized for severeLIS patients because they require the use of a part of theiraffected muscles, such as eye movements for controllingsuch exogenous brain switches [47]. In contrast, endogenousbrain switches use self-modulated brain activity, such as SMRand hemodynamic responses, to generate discriminative brainpatterns. In endogenous brain switches, a user performs a des-ignated mental task to modulate a task-specific brain activity,and a brain switch detects the self-modulated brain activityto turn on the BCI system. The absence of external stimuliis advantageous because the mentioned problematic issuescaused by the use of external stimuli can be prevented. Despitethis advantage of endogenous brain switches, it has beenless in the focus of research because of comparatively lowerperformance to exogenous brain switches. Thus, it would behighly useful to improve the performance of endogenous brainswitches.

Recently, hybrid EEG/NIRS BCI systems, which simul-taneously use EEG and NIRS, have been introduced toimprove the performance of unimodal BCI systems based onendogenous paradigms [1], [48]–[53]. In general, hybrid BCIsextract discriminant features from each modality, and mergetheir information in the procedure of pattern classification[51], [54]. This classification framework based on multi-modalities has successfully shown the superior classificationperformance compared to that of a unimodal classificationframework based on EEG or NIRS in endogenous BCIs[1], [48]–[51]. Despite the advantage of hybrid EEG/NIRSBCIs in terms of classification performance, to the best ofour knowledge, it has been not evaluated whether a hybridEEG/NIRS modality can improve the onset detection per-formance of brain switches developed based on endogenousparadigms.

Fig. 1. Schematic diagram describing the experimental paradigmused for recording the dataset B in the open-access dataset for hybridEEG/NIRS BCI studies [1].

In this study, we therefore focus on this important aspect inBCI research and developed an endogenous brain switch basedon the simultaneous use of EEG and NIRS data. We comparedthe performance of a hybrid EEG/NIRS brain switch with uni-modal EEG- and NIRS-based brain switches in terms of TPR,FPR, onset detection time (ODT), and information transferrate (ITR). For this objective, an open-access dataset was usedthat includes hybrid EEG/NIRS data measured while twenty-nine subjects performed mental arithmetic (MA) task [1].Additionally, a pseudo-online simulation was conducted toconfirm the feasibility of implementing an online BCI systemwith our hybrid EEG/NIRS brain switch. The rest of this paperis organized as follows. Methods and corresponding results areexplained in section II and III, respectively. Discussion andconclusion are presented in section IV.

II. METHODS

A. EEG/NIRS DatasetAn open-access dataset for hybrid EEG/NIRS BCI studies

was used for this study [1]. This dataset consisted of twodifferent datasets: i) dataset A (left- versus right-hand motorimagery) and ii) dataset B (MA versus REST task). Dataset Bwas only used for this study because we focused on detectingthe onset of a specific mental task as compared to the restingstate. In dataset B, twenty-nine healthy subjects repeatedlyperformed an MA task that is continuous subtraction ofa one-digit number from a three-digit number sequentially,e.g., 384-8, and took a rest, while EEG and NIRS signalswere simultaneously measured. Fig. 1 shows the schematicdiagram of the experimental paradigm used for recording thedataset B. An experimental session consisted of a 60-s pre-rest period, 20 repetitions of the given task, and a 60-s post-rest period. Each task started with a 2-s visual instruction forthe preparation of the given task, followed by a 10-s taskperiod (either MA or REST), after which a resting period wasgiven between 15 to 17 s. During the task and resting period,a fixation cross was provided to each subject to minimize EOGartifacts, such as eye blinks and movements. A single MA trialconsisted of a 10-s task and a 10-s rest, while a REST trial is



composed of a 20-s rest (10-s task and 10-s rest). Each subjectcompleted three sessions, resulting in 30 MA and 30 RESTtrials for each subject.

EEG data were recorded on thirty electrodes according tothe international 10-5 system (AFp1, AFp2, AFF1h, AFF2h,AFF5h, AFF6h, F3, F4, F7, F8, FCC3h, FCC4h, FCC5h,T7, T8, Cz, CCP3h, CCP4h, CCP5h, CCP6h, Pz, P3, P4,P7, P8, PPO1h, PPO2h, POO1, POO2, and Fz). NIRSdata were measured on thirty-six NIRS channels made byfourteen sources and sixteen detectors (AF7Fp1, AF3Fp1,AF3AFz, FpzFp1, FpzAFz, FpzFp2, AF4AFz, AF4Fp2,AF8Fp2, C5CP5, C5FC5, C5C3, FC3FC5, FC3C3, FC3FC1,CP3CP5, CP3C3, CP3CP1, C1C3, C1FC1, C1CP1, C3FC2,C2CP2, C2C4, FC4FC2, FC4C4, FC4FC6, CP4CP6, CP4CP2,CP4C4, CP6P6, C6C4, C6FC6, OzPOz, OzO1, and OzO2).There were nine NIRS channels around Fp1, Fp2, and Fpz,twenty-four NIRS channels around C3 and C4, and three NIRSchannels around Oz. The sampling frequency rates of EEG andNIRS data were 200 and 12.5 Hz, respectively. More detailedinformation about the dataset is provided in [1].

In this study, we assumed an MA task is a means of a brainswitch in an idle state to turn on an asynchronous BCI systemwhile resting state is a means of a brain switch to stay in anidle state; MA is a task for brain switch which plays a role toconvert an idle state into a control state in an asynchronousBCI system.

B. Preprocessing of EEG and NIRS Data

The raw EEG and NIRS signals were preprocessed by aseries of signal processing algorithms to remove artifacts. Alldata processing was done using MATLAB R2016b (Math-Works, Natick, MA, USA). The raw EEG data were firstre-referenced using a common average reference (CAR) andfiltered using a fourth-order Chebyshev type II filter witha passband of 0.5-50 Hz. Independent component analy-sis (ICA)-based electrooculography (EOG) artifact rejectionwas conducted using a multiple artifact rejection algorithm(MARA) in the EEGLAB toolbox [55], [56]. After removalof EOG artifacts, EEG epochs were extracted from −5 to20 s based on task onset time, and baseline correction wasperformed using the value averaged between -5 and -2 s.EEG data measured on ten frontal channels (AFp1, AFp2,AFF5h, AFF1h, AFF2h, AFF6h, F7, F3, F4, and F8) wereonly selected for further analysis because frontal areas aremost associated with the MA task [57]–[60], and employ-ing only frontal areas is more practical to implement aBCI system. The NIRS data were band-pass filtered usinga sixth-order zero-phase Butterworth filter with a passbandof 0.01-0.1 Hz to reduce physiological noises and low-frequency drifts [13], [20], [61]–[64]. The NIRS data measuredon the frontal area were also selected for further analysis asfor EEG data [65]–[67]. NIRS epochs from −5 to 20 s wereequally extracted as EEG epochs, and the values averagedbetween −5 and −2 s were used for baseline correction.C. Procedure of Onset Detection

Features for EEG and NIRS were extracted to trainEEG, NIRS, and hybrid EEG/NIRS classifiers used for onset

detection in a brain switch. A common spatial pattern (CSP)analysis with auto-filter selection was used to extract EEGfeatures using the BBCI toolbox [68], which is a popularEEG data analysis tool. Different mental imagery tasks showthe distinct spatial distributions of EEG signals. The CSPalgorithm is a spatial filtering method to maximize the dif-ference of EEG spatial distribution for each imagery taskin terms of variance. Thus, different imagery tasks can beclassified using features extracted by the CSP filters. ForCSPs, subject-specific band-pass filter coefficients were esti-mated by the means of a commonly used heuristic proce-dure which estimates a frequency band showing the highestabsolute signed squared point biserial correlation coefficient(signed r2) value [69], [70]. The EEG data were band-passfiltered using the subject-specific band-pass filter coefficients,and then log-variances of the first and last two CSP com-ponents were obtained as EEG features. Means and slopes ofoxygenated (HbO) and deoxygenated hemoglobin (HbR) werecalculated as NIRS features, which were noted as promisingfeature candidates in previous NIRS studies [20], [71], [72].

Using the EEG and NIRS features, we estimated the onsetdetection performance of EEG, NIRS, and hybrid EEG/NIRSbrain switches using six-fold cross-validation. One-fold con-sisted of ten trials (five trials for MA and the other fivetrials for REST). For each iteration, five-folds and one-foldwere used as training data and test data, respectively. Tworegularized linear discriminant analysis (rLDA) classifiers withautomatic shrinkage selection [73] were trained for each ofEEG and NIRS data using training trials between 0 and 10 s(task period). A hybrid rLDA classifier was trained by usingthe rLDA outputs of the EEG and NIRS classifiers [51].An rLDA classifier was selected for this study because theclassification performance of the rLDA classifier was highestcompared to other classifiers, such as LDA, quadratic DA,support vector machine (SVM), and random forest classifiers(detailed results are not shown). To estimate the classificationperformance, each trial in the test data was segmented usinga 2-s moving window with 50 % overlap, and the segmentedepochs were sequentially tested along the time. The entire trialperiod from 0 to 20 s (10-s task period and 10-s rest period)was used to investigate the overall trend of the classificationperformance as in the previous study [1] that provided the dataused in this study. However, only the 10-s task period (the first10-s of MA and REST task) was used to estimate the onsetdetection performance of a brain-switch based on a templatematching algorithm.

Each of EEG, NIRS, and hybrid EEG/NIRS classifierswere used in corresponding brain switches. There were ninesegments created by a 2-s moving window with 50% overlapin a 10-s test trial. Each segment was sequentially testedfor each of the three classifiers independently. Consequently,each classifier produced nine classification results: 0 (REST)or 1 (MA). Onset intention was detected using a templatematching algorithm [31], [74]. The algorithm judges that anonset intention is detected when a classifier gives same tasklabels consecutively as much as a predefined template size. Forexample, if a predefined template size is three, the algorithmjudges that an onset is detected when a classifier produces



Fig. 2. Example of the pseudo-online simulation when the number oftraining trials is ten (5 MA and 5 REST).

the same outputs consecutively three times in a row. Someexamples are as follows (0: REST, 1: MA): 010111, 01100111,and 0111 for MA onset detection (true positive) while 01000,11000, and 0101000 for REST onset detection (true negative).An optimal template size was selected considering the perfor-mance measures of onset detection.

D. Pseudo-Online Simulation

A pseudo-online simulation was performed to examine thefeasibility of implementing an online BCI system using ourhybrid EEG/NIRS brain switch. A training dataset was madeusing continuous trials for each subject. The performance ofthe pseudo-online simulation was evaluated using the differentsize of the training dataset from five to fifty with five spanlengths (e.g. 5, 10, 15, 20, …, 50) to see the impact of theamount of training dataset on onset detection performance.Note that the number of trials is sixty for each subject (30 MAand 30 REST). For example, when the size of the trainingdataset was ten, trials from first to the tenth were used asa training dataset while the other fifty trials from eleventhto sixtieth were used as a test dataset (Fig. 2). Thus, a testdataset did not contain training trials to mimic a real onlineBCI scenario. We fixed template sizes used in the templatematching algorithm at three based on the offline analysisresults that will be presented in the results section.

E. Performance Measures of Onset Detection

To compare the onset detection performances of EEG,NIRS, and hybrid EEG/NIRS brain switches, we calculatedfour different measures: TPR, FPR, ODT, and ITR. In thisstudy, we define as true positive when MA was correctlydetected within the 10-s task period and as true negative whenREST was correctly detected within the first 10-s period of theREST trial. Therefore, TPR indicates the ratio of the numberof correctly detected trials to the total number of MA trialswithin a task period of 10 s. On the other hand, FPR is the ratioof the number of incorrectly detected trials to the total numberof REST trials within the first 10-s period. Units of TPR andFPR were a percentage and the number of false positives perminute, respectively, as described in previous studies [36],[75], [76]. ODT was calculated by estimating the time whenthe onset was detected by the template matching algorithmfor each trial. Finally, ITR defined in bits per minute wascalculated by the number of trials per minute (V ), the numberof available commands (N), and classification accuracy (P)as follows [3], [4].

ITR = V ∗ (log2 N + P ∗ log2 P+ (1 − P) ∗ log21 − PN − 1 )

(1)

Please note that V was calculated by equation (2).

V = (60/O DT ) − F (2)F was the number of false positives per minute. The FPRwas regarded as a penalty because additional commands arerequired if false operations are generated in asynchronous BCIparadigms.

F. Statistical Analysis

A statistical analysis was performed to confirm the sig-nificant difference between the performances of EEG, NIRS,and hybrid EEG/NIRS brain switches. A same procedure wasused for all statistical tests in this study. Firstly, the Shapiro-Francia test was performed to test the normality of statisticaldatasets [77]. Analysis of variance (ANOVA) was performedwhen the data followed normality while the Friedman test wasperformed when the data did not follow normality. In ANOVA,Levene’s test was additionally applied to check whether sta-tistical datasets have equal variances or not [78]. If statisticaldatasets have unequal variances, Welch ANOVA was used tocorrect the unequal variances. Secondly, t-test or Wilcoxonsigned-rank test was used for ANOVA and Friedman test asposthoc analysis, respectively. Finally, Bonferroni correctionwas conducted to calculate corrected p-values, and the signif-icance level was set as p < 0.05.

III. RESULTS

A. Classification Performances of EEG, NIRS, andHybrid EEG/NIRS Classifiers

This section shows the overall trend of classification resultsof the three classifiers (EEG, NIRS, and hybrid EEG/NIRS),and the performance of the respective brain switches willfollow later. Fig. 3 shows the classification performances ofEEG, NIRS, and hybrid EEG/NIRS classifiers along the timeperiod created by a 2-s moving window with 50% over-lap. Overall, the hybrid EEG/NIRS classifier shows signif-icantly higher accuracies than those of unimodal EEG andNIRS classifiers at most time periods in the task periodof 0–10 s (Fig. 3(a)). Red and blue circles in Fig. 3 indi-cate that the accuracy of a hybrid EEG/NIRS classifier issignificantly higher than those of NIRS or EEG classifiers,respectively, at the corresponding time periods. The maximumclassification accuracies of EEG, NIRS, and hybrid EEG/NIRSclassifiers are 74.43 % at a time period of 4-6 s, 73.91 %at a time period of 8-10 s, and 78.62 % at a time periodof 8-10 s, respectively. The classification accuracy of thehybrid EEG/NIRS classifier significantly decreases after 10 s,and it is lower than that of the NIRS classifier. The reasonwas that trials from 0 to 10 s were used to train classifiers.Because EEG has a higher temporal resolution than NIRS [79],discriminative EEG patterns disappear as soon as the task isfinished. On the other hand, NIRS has an inherent time-delayof several seconds until task-specific hemodynamic responsesappear and disappear [14]. Thus, the classification accuracyof the NIRS classifier can be maintained by this effect eventhough the task was in fact already completed. The overalltrend of the classification result coincides with that of previoushybrid BCI studies [49], [50].



Fig. 3. Classification performances of EEG, NIRS, and hybrid EEG/NIRS classifiers along the time periods created by a 2-s moving window with50% overlap: (a) classification accuracy, (b) sensitivity, and (c) specificity. Shaded areas indicate the standard errors of classification accuracies,sensitivities, and specificities at each time point. Gray dotted vertical lines mean the end of the task. Red and blue circles show that the classificationaccuracy of a hybrid EEG/NIRS classifier is significantly higher than those of a NIRS or EEG classifier, respectively.

Fig. 4. Scalp topographies of the separability between mental arithmetic (MA) and resting state (REST) for EEG and NIRS features in terms ofsigned r2. A higher singed r2 value indicates a better separability between the two conditions (MA vs. REST) regardless of its sign. Individual signedr2values are estimated and averaged across all subjects. A gray vertical dotted line indicates the task onset. Numbers in the bottom of each panelrepresent time periods based on the task onset.

B. Separability Between MA and REST for EEG andNIRS Features

Fig. 4 shows the scalp topographies of the separabilitybetween MA and REST for EEG and NIRS features, respec-tively, in terms of signed squared pointwise biserial correlationcoefficient values (signed r2) at specific time periods. Notethat a higher value of singed r2 indicates a better separabilitybetween the two conditions, regardless of its sign. EEGtopographies show the high separability from the task onsetand last the phenomenon until the end of the task; whereasNIRS topographies show the high separability about 3–5 safter the task starts due to the inherently delayed hemodynamic

response. Note that the signs of r2 values for HbO andHbR are inverted after task-related hemodynamic responsesappear at a time period of 3–5 s, which is a well-documentedneurophysiological phenomenon. These topographical resultssupport that the MA task can be effectively classified with theREST task and can be used as a means for the developmentof an endogenous brain switch.

C. Onset Detection Performances of EEG, NIRS, andHybrid EEG/NIRS Brain Switches

Fig. 5 shows the receiver operation characteristic (ROC)curves for EEG, NIRS, and hybrid EEG/NIRS brain switches.



Fig. 5. Receiver operating characteristic (ROC) curves of EEG, NIRS,and hybrid EEG/NIRS brain switches. Each number indicates the size ofthe template used for a template matching algorithm.

Fig. 6. Area under the curves (AUCs) of EEG, NIRS, and hybridEEG/NIRS brain switches. The AUC of a hybrid EEG/NIRS brain switchis significantly higher than those of EEG and NIRS brain switches(corrected p < 0.05).

The TPRs and FPRs were calculated as a function of thetemplate size used in the template matching algorithm. Thehybrid EEG/NIRS brain switch shows the most reliable onsetdetection performance for all template sizes. The area underthe curve (AUC) of the hybrid EEG/NIRS brain switch islarger than those of EEG and NIRS brain switches. Thequantitative results of the AUCs are shown in Fig. 6; the AUCof the hybrid EEG/NIRS brain switch is significantly higherthan those of EEG and NIRS brain switches.

The detailed results of TPR, FPR, and ODT, and ITR areshown in Table I as a function of template size. In line withthe results of Figs. 5 and 6, the hybrid EEG/NIRS brain switchshows higher TPRs for all template sizes and lower FPRs thanthose of EEG and NIRS brain switches when a template sizeis more than three. For ODT, the EEG brain switch generallyshows better performance as compared to NIRS and hybridEEG/NIRS brain switches, but little difference between EEGand hybrid EEG/NIRS brain switches is observed withoutstatistical significance. The ITRs of the hybrid EEG/NIRS

Fig. 7. Statistical results of four performance measures (true positiverate, false positive rate, onset detection time, information transfer rate)for each brain switch when the template size is fixed as three. Asterisksindicate significant differences between corresponding pairs.

brain switch were generally higher than those of EEG andNIRS brain switches. Fig. 7 shows the statistical results offour performance measures (TPR, FPR, ODT, and ITR) foreach brain switch when the template size was fixed as three.The parameter three was decided as an optimal template sizebecause we could get the acceptable performance (relativelyhigh TPR, low FPR, fast ODT, and high ITR), and a similartrend was obtained between the performances of EEG, NIRS,and hybrid EEG/NIRS brain switches when a template size ismore than three, as shown in Table I. The hybrid EEG/NIRSbrain switch shows a significantly better performance than theNIRS brain switch for all the four performance measures, andthen the EEG brain switch for FPR. No significant difference isobserved between EEG and hybrid EEG/NIRS brain switchesfor TPR, ODT, and ITR.

D. Results of Pseudo-Online Simulation

Figs. 8 shows the results of the pseudo-online simulationfor EEG, NIRS, and hybrid EEG/NIRS brain switches, respec-tively. Red circles in Figs. 8 indicate that the performance of ahybrid EEG/NIRS brain switch is significantly better than thatof NIRS brain switch, respectively, at the corresponding con-dition. The overall trend of pseudo-online simulation resultsis in line with that of offline analysis results even thoughthe performances of three brain switches vary, depending onthe amount of training data. The EEG and hybrid EEG/NIRSbrain switch show significantly better TPR, ODT, and ITRthan those of the unimodal NIRS brain switch in general(Fig. 8). However, no significant difference between threebrain switches is observed for FPR even though the hybridEEG/NIRS brain switch shows a better FPR performance thanunimodal EEG and NIRS brain switches.

Fig. 9 shows an example of the comparison of the offlineand pseudo-online performances for EEG, NIRS, and hybridEEG/NIRS brain switches when the amount of training datais same (fifty trials). Note that because we had a total ofsixty trials (30 MA and 30 REST) and performed a six-fold



TABLE IONSET DETECTION PERFORMANCES OF NIRS, EEG, AND HYBRID EEG/NIRS BRAIN SWITCHES

WITH RESPECT TO THE TEMPLATE SIZE USED IN A TEMPLATE MATCHING ALGORITHM

Fig. 8. Pseudo-online simulation results of EEG, NIRS, and hybrid EEG/NIRS brain switches. Colorful circles show significant differences betweencorresponding pairs, such as hybrid vs NIRS, EEG vs NIRS, and EEG vs hybrid. Vertical lines mean standard errors.

cross validation, fifty trials (25 MA and 25 REST) wereused as training data for each iteration in the offline analysis.Except that an EEG brain switch shows a significant differ-ence between offline and pseudo-online performance for ITR,no significant difference between the offline and pseudo onlineanalysis schemes is observed in all performance measures.

IV. DISCUSSION

A brain switch is at the core of a two-step approach todevelop asynchronous BCIs, aiming at reducing the FPR ofasynchronous BCIs. In this study, we proposed an endogenous

brain switch based on the simultaneous use of EEG andNIRS data to improve the onset detection performance of anendogenous brain switch, thereby developing a more reliableasynchronous BCI system. The onset detection performanceof a hybrid EEG/NIRS brain switch was compared with thatof an EEG and NIRS brain switch to check the feasibility ofa hybrid EEG/NIRS brain switch.

It was observed from the offline data analysis that a hybridEEG/NIRS brain switch generally showed a significantlybetter onset detection performance than a unimodal EEGand NIRS brain switch. When an optimal template size was



Fig. 9. Performance comparison of the offline analysis and the pseudo-online simulation analysis for EEG, NIRS, and hybrid EEG/NIRS brainswitches. The template size was three, and the number of training trials for the offline and pseudo-online analysis was fifty. Vertical lines meanstandard errors. An asterisk indicates that the pseudo-online ITR of the EEG brain switch is significantly higher than the offline ITR of the EEG brainswitch.

selected (three), a hybrid EEG/NIRS brain switch showed asignificantly better performance than a NIRS brain switchfor all performance measures (TPR, FPR, ODT, and ITR).However, only FPR showed a significantly better perfor-mance for a hybrid EEG/NIRS brain switch than an EEGbrain switch. There was no statistically significant differencebetween a hybrid EEG/NIRS and an EEG brain switch forthe other three performance measures (TPR, ODT, and ITR).Even though the significant difference between a hybridEEG/NIRS and an EEG brain switch is shown for FPR, thisresult is still practically helpful because FPR was significantlyreduced when a hybrid EEG/NIRS brain switch was used.As aforementioned, FPR is of special interest in a brain switchbecause once it occurs, further correcting commands would berequired. Most importantly, a hybrid EEG/NIRS brain switchshowed a significantly higher AUC than an EEG brain switch,as shown in Figs. 5 and 6. This means that the overall onsetperformance of a hybrid EEG/NIRS brain switch is better thanthat of an EEG brain switch regardless of the template sizein terms of TPR and FPR. Taken the offline analysis resultstogether, it can be concluded that hybridizing EEG/NIRSdata can enhance the onset detection performance of a brainswitch.

The feasibility of the offline analysis results was addi-tionally confirmed by a pseudo-online simulation analysis.The overall tendency of the pseudo-online simulation resultswas in line with the offline analysis results, except that ahybrid EEG/NIRS brain switch does not show a significantlybetter performance than an EEG brain switch for FPR eventhough the FPRs of a hybrid EEG/NIRS brain switch werebetter than those of an EEG brain switch regardless ofthe amount of training data. As the offline analysis results,a hybrid EEG/NIRS brain switch generally showed a signifi-cantly higher performance than a NIRS brain switch for TPR,ODT, and ITR. In the comparison of the offline and pseudo-online performances for EEG, NIRS, and hybrid EEG/NIRS

brain switches (Fig. 9), no significant difference between theoffline and pseudo online analysis schemes was observed inall performance measures, except that an EEG brain switchshowed a significant difference between offline and pseudo-online performance for ITR. These simulated online analysisresults demonstrated the possibility that the offline analysisresults obtained in this study can indeed be transferred to areal online scenario. However, real online experiments shouldfollow to carefully address the practical feasibility of theproposed hybrid EEG/NIRS brain switch.

There are two different types of brain switches: exoge-nous and endogenous brain switches. According to previ-ous studies, exogenous brain switches have shown betteronset detection performances than those of endogenous brainswitches [31]. Interestingly, the onset detection performance ofour hybrid EEG/NIRS brain switch based on endogenous par-adigm was comparable to those of previous exogenous brainswitches. Our hybrid EEG/NIRS brain switch showed a TPRof 87.82 ± 13.92 % and an FPR of 0.88 ± 0.43 FPs/minwhen the template size was predefined as three. Zhang et al.introduced a brain switch based on P300, and its FPR was0.71 FPs/min [75]. Also, SSVEP-based brain switches exhib-ited an FPR of 0.38 FPs/min and a TPR of 100.00 % inLim et al.’s study [31], and an FPR of 0.98 FPs/min andTPR of 78.75% [36]. A hybrid brain switch, which simulta-neously uses SSVEP and P300, was introduced by Peng et al.in 2016 [80], and the hybrid brain switch showed a TPRof 93.75 % and an FPR of 0.15 FPs/min. In addition tothe EEG-based brain switches, an NIRS-based brain switchwas developed, and it showed a TPR of 88.00 % and anFPR of 0.04 FPs/min [43]. Even though the mentioned threebrain switches showed relatively higher TPRs and lower FPRsthan those of this study, our study showed a much fasterODT (4.88±0.49 sec) than those of the previous brain-switchstudies [31], [43], [80]. These results demonstrate that themultimodal approach simultaneously using EEG and NIRS



could effectively improve the onset detection performance ofendogenous brain switches.

Most of the previous hybrid EEG/NIRS BCI studies havefocused on the development of synchronous BCI systems,where EEG and NIRS were simultaneously used to improveBCI performance [1], [49]–[51]. A few studies have attemptedto develop asynchronous BCI systems based on a hybridapproach, but they did not use EEG and NIRS data simul-taneously. Instead, NIRS data was used for detecting onsetintention in an idle state while EEG data was used for discrim-inating multiple control intentions in a control state [41]–[43],[81]; SSVEP-[41], [42] or SMR-[43], [81] based BCI systemsare operated using EEG data after a control state is turned onby a NIRS brain switch. Therefore, the feasibility of mergingdiscriminative features of EEG and NIRS to detect a taskonset has been unknown. In this study, we investigated whethersimultaneous use of EEG and NIRS data can be useful whendeveloping a brain switch of an asynchronous BCI system. Ourexperimental results indicate that the hybrid EEG/NIRS brainswitch can provide more reliable onset detection performancesthan a unimodal EEG or NIRS brain switch. To best of ourknowledge, this is the first study reporting the onset detectionperformance of a hybrid EEG/NIRS brain switch.

In this study, a simple template matching algorithm wasused for onset detection [31], [74]. Similar to the templatematching algorithm, most of the previous brain switch studieshave used simple thresholding methods for onset detections[29], [30], [36], [43], [75], and thus the development of anovel algorithm is required to further increase the perfor-mance of a brain switch. Using advanced detection algorithmsbased on deep learning techniques [82]–[84], threshold-freemethods [32], and dynamic-stopping approaches [85] can bepotentially considerable options to obtain a better onset detec-tion performance than template matching and thresholdingmethods. In our future studies, they will be considered tofurther improve the onset detection performance of a hybridEEG/NIRS brain switch, thereby to contribute to the develop-ment of reliable asynchronous BCI systems.

In this study, we used the open access EEG/NIRS hybriddataset that contained MA and REST trials independently [1].We defined true positive when MA was correctly detectedwithin the 10-s task period and true negative when REST wascorrectly detected within the first 10-s period of the RESTtrial. Even though a continuous analysis between differenttasks (i.e., MA and REST) would have been ultimately helpfulto accurately investigate the performance of a brain-switch,only a continuous analysis could be employed within eachtask in this study due to the configuration of the analyzedopen-access dataset. A task period was fixed as 10 s for MAwhen acquiring the open-access dataset, but onset detectiontime was about 5 s on average when using an optimal templatesize of three (Table I). For a continuous analysis between MAand REST, the data directly after MA onset detection shouldbe the one induced in resting state, but the rest of the dataafter MA onset detection is as well induced by MA untilthe end of the task, which unfortunately makes a continuousanalysis unsuitable using the open-access data used in thisstudy. Moreover, there are different types of resting-state

data: i) pure resting-state data (REST) and ii) resting-statedata right after the MA period, including a refractory periodin the first several second period (e.g., 5 s for EEG [32],[38], [86]–[88]), which is a transition period from MA toresting state. The two different types of resting-state data alsohave a different characteristics [89], which also makes a con-tinuous analysis unsuitable regardless of whether either typeis used as training data because the characteristics of trainingand test data are generically different. To resolve the issuerelated to the refractory period, optimization of the refractoryperiod would in principle be required accompanying a morecomplicated classification procedure, which is complicated inits own right and deserves an own comprehensive analysis.This would be beyond the scope of this study aiming atinvestigating the feasibility of using a hybrid approach on theperformance improvement of task onset detection. In previousbrain-switch studies, the classification outputs of the refractoryperiod are generally ignored to prevent false operations and therefractory period is kept until the complete extinction of thetask-specific brain activity sustained even after a mental task(e.g., MA in this study) [32], [38], [86]–[88]. Based on theresults shown in Fig. 3, the NIRS data would have a longerrefractory period than the EEG data due to the inherentlydelayed hemodynamic response, and thus a solution to resolvethe problem regarding the refractory period would be morecomplicated in EEG/NIRS hybrid approaches. This problemof the refractory period should in the future be solved in amore intelligent way, not just by ignoring a certain periodright after task onset detection, to develop a practical brain-switch. Because a continuous analysis reflects a real brain-switch scenario, the related limitation should be addressed infuture studies by obtaining a dedicated more realistic, contin-uous EEG/NIRS data to further demonstrate the practicality ofour proposed hybrid EEG/NIRS brain-switch. Even though acontinuous analysis was not performed due to the mentionedissues, we believe that our study could contribute to the BCIcommunity in that the feasibility of using an EEG/NIRS hybridapproach was proved on the performance improvement of abrain-switch, to the best of our knowledge, for the first time.

REFERENCES

[1] J. Shin et al., “Open access dataset for EEG+NIRS single-trial clas-sification,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 25, no. 10,pp. 1735–1745, Oct. 2017.

[2] J. R. Wolpaw et al., “Brain-computer interface technology: A review ofthe first international meeting,” IEEE Trans. Rehabil. Eng., vol. 8, no. 2,pp. 164–173, Jun. 2000.

[3] J. Wolpaw, N. Birbaumer, D. McFarland, G. Pfurtscheller, andT. Vaughan, “Brain-computer interfaces for communication and control,”Clin. Neurophys., vol. 113, no. 6, pp. 767–791, 2002.

[4] G. Dornhege, J. d. R. Millan, T. Hinterberger, D. J. McFarland, andK.-R. Müller, Toward Brain-Computer Interfacing. Cambridge, MA,USA: MIT Press, 2007.

[5] H.-J. Hwang, J.-H. Lim, Y.-J. Jung, H. Choi, S. W. Lee, and C.-H. Im,“Development of an SSVEP-based BCI spelling system adopting aQWERTY-style LED keyboard,” J. Neurosci. Methods, vol. 208, no. 1,pp. 59–65, Jun. 2012.

[6] D.-W. Kim, J.-H. Cho, H.-J. Hwang, J.-H. Lim, and C.-H. Im, “A vision-free brain-computer interface (BCI) paradigm based on auditory selec-tive attention,” in Proc. Annu. Int. Conf. IEEE Eng. Med. Biol. Soc.,Aug. 2011, pp. 3684–3687.

[7] C. Guger et al., “How many people are able to control a P300-based brain–computer interface (BCI)?” Neurosci. Lett., vol. 462, no. 1,pp. 94–98, Sep. 2009.



[8] G. Pfurtscheller and C. Neuper, “Motor imagery and direct brain-computer communication,” Proc. IEEE, vol. 89, no. 7, pp. 1123–1134,Jul. 2001.

[9] J. Mellinger et al., “An MEG-based brain-computer interface (BCI),”NeuroImage, vol. 36, no. 3, pp. 581–593, 2007.

[10] L. Kauhanen et al., “EEG and MEG brain–computer interface fortetraplegic patients,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 14,no. 2, pp. 190–193, Jun. 2006.

[11] R. Sitaram et al., “Temporal classification of multichannel near-infraredspectroscopy signals of motor imagery for developing a brain–computerinterface,” NeuroImage, vol. 34, no. 4, pp. 1416–1427, Feb. 2007.

[12] J. Shin, K.-R. Müller, and H.-J. Hwang, “Near-infrared spectroscopy(NIRS)-based eyes-closed brain-computer interface (BCI) using pre-frontal cortex activation due to mental arithmetic,” Sci. Rep., vol. 6,no. 1, p. 36203, Dec. 2016.

[13] H.-J. Hwang, J.-H. Lim, D.-W. Kim, and C.-H. Im, “Evaluation ofvarious mental task combinations for near-infrared spectroscopy-basedbrain-computer interfaces,” J. Biomed. Opt., vol. 19, no. 7, Jul. 2014,Art. no. 077005.

[14] S. M. Coyle, T. E. Ward, and C. M. Markham, “Brain-computer inter-face using a simplified functional near-infrared spectroscopy system,”J. Neural Eng., vol. 4, no. 3, p. 219, 2007.

[15] N. Weiskopf et al., “Principles of a brain-computer interface (BCI)based on real-time functional magnetic resonance imaging (fMRI),”IEEE Trans. Biomed. Eng., vol. 51, no. 6, pp. 966–970, Jun. 2004.

[16] R. Sitaram et al., “FMRI brain-computer interface: A tool for neuro-scientific research and treatment,” Comput. Intell. Neurosci., vol. 2007,pp. 1–10, Jan. 2007.

[17] S.-S. Yoo et al., “Brain–computer interface using fMRI: Spatial nav-igation by thoughts,” NeuroReport, vol. 15, no. 10, pp. 1591–1595,Jul. 2004.

[18] R. Leeb, D. Friedman, G. R. Müller-Putz, R. Scherer, M. Slater, andG. Pfurtscheller, “Self-paced (asynchronous) BCI control of a wheel-chair in virtual environments: A case study with a tetraplegic,” Comput.Intell. Neurosci., vol. 2007, pp. 1–8, Jan. 2007.

[19] J. del R. Millan and J. Mouriño, “Asynchronous BCI and local neuralclassifiers: An overview of the adaptive brain interface project,” IEEETrans. Neural Syst. Rehabil. Eng., vol. 11, no. 2, pp. 159–161, Jun. 2003.

[20] S. D. Power, A. Kushki, and T. Chau, “Towards a system-pacednear-infrared spectroscopy brain–computer interface: Differentiating pre-frontal activity due to mental arithmetic and mental singing from the no-control state,” J. Neural Eng., vol. 8, no. 6, Oct. 2011, Art. no. 066004.

[21] B. Blankertz, G. Dornhege, M. Krauledat, K.-R. Müller, and G. Curio,“The non-invasive Berlin brain–computer interface: Fast acquisition ofeffective performance in untrained subjects,” NeuroImage, vol. 37, no. 2,pp. 539–550, Aug. 2007.

[22] B. Blankertz, G. Curio, and K.-R. Müller, “Classifying single trial EEG:Towards brain computer interfacing,” in Proc. Adv. Neural Inf. Process.Syst., 2002, pp. 157–164.

[23] C.-H. Han, E. Kim, and C.-H. Im, “Development of a brain–computerinterface toggle switch with low false-positive rate using respiration-modulated photoplethysmography,” Sensors, vol. 20, no. 2, p. 348,Jan. 2020.

[24] B. Xia, X. Li, H. Xie, W. Yang, J. Li, and L. He, “Asynchronous brain-computer interface based on steady-state visual-evoked potential,” Cogn.Comput., vol. 5, no. 2, pp. 243–251, 2013.

[25] R. Liu, Y.-X. Wang, and L. Zhang, “An FDES-based shared controlmethod for asynchronous brain-actuated robot,” IEEE Trans. Cybern.,vol. 46, no. 6, pp. 1452–1462, Jun. 2016.

[26] R. Zhang et al., “Control of a wheelchair in an indoor environ-ment based on a brain–computer interface and automated navigation,”IEEE Trans. Neural Syst. Rehabil. Eng., vol. 24, no. 1, pp. 128–139,Jan. 2016.

[27] I. Volosyak, F. Gembler, and P. Stawicki, “Age-related differences inSSVEP-based BCI performance,” Neurocomputing, vol. 250, pp. 57–64,Aug. 2017.

[28] J. F. Borisoff, S. G. Mason, A. Bashashati, and G. E. Birch, “Brain–Computer interface design for asynchronous control applications:Improvements to the LF-ASD asynchronous brain switch,” IEEE Trans.Biomed. Eng., vol. 51, no. 6, pp. 985–992, Jun. 2004.

[29] G. Pfurtscheller, T. Solis-Escalante, R. Ortner, P. Linortner, andG. R. Muller-Putz, “Self-paced operation of an SSVEP-based orthosiswith and without an imagery-based ‘brain switch’: A feasibility studytowards a hybrid BCI,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 18,no. 4, pp. 409–414, Aug. 2010.

[30] G. R. Müller-Putz, V. Kaiser, T. Solis-Escalante, and G. Pfurtscheller,“Fast set-up asynchronous brain-switch based on detection of foot motorimagery in 1-channel EEG,” Med. Biol. Eng. Comput., vol. 48, no. 3,pp. 229–233, Mar. 2010.

[31] J.-H. Lim et al., “An emergency call system for patients in locked-instate using an SSVEP-based brain switch,” Psychophysiology, vol. 54,no. 11, pp. 1632–1643, Nov. 2017.

[32] S. He et al., “A P300-based threshold-free brain switch and its appli-cation in wheelchair control,” IEEE Trans. Neural Syst. Rehabil. Eng.,vol. 25, no. 6, pp. 715–725, Jun. 2017.

[33] R. Xu et al., “Endogenous sensory discrimination and selection by afast brain switch for a high transfer rate brain-computer interface,”IEEE Trans. Neural Syst. Rehabil. Eng., vol. 24, no. 8, pp. 901–910,Aug. 2016.

[34] B. Blankertz et al., “Boosting bit rates and error detection for theclassification of fast-paced motor commands based on single-trial EEGanalysis,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 11, no. 2,pp. 127–131, Jun. 2003.

[35] C.-H. Han, K.-R. Müller, and H.-J. Hwang, “Brain-switches for asyn-chronous brain–computer interfaces: A systematic review,” Electronics,vol. 9, no. 3, p. 422, Mar. 2020.

[36] J. Pan, Y. Li, R. Zhang, Z. Gu, and F. Li, “Discrimination betweencontrol and idle states in asynchronous SSVEP-based brain switches: Apseudo-key-based approach,” IEEE Trans. Neural Syst. Rehabil. Eng.,vol. 21, no. 3, pp. 435–443, May 2013.

[37] R. Ortner, B. Z. Allison, G. Korisek, H. Gaggl, and G. Pfurtscheller, “AnSSVEP BCI to control a hand orthosis for persons with tetraplegia,”IEEE Trans. Neural Syst. Rehabil. Eng., vol. 19, no. 1, pp. 1–5,Feb. 2011.

[38] G. Pfurtscheller and T. Solis-Escalante, “Could the beta rebound inthe EEG be suitable to realize a ‘brain switch’?” Clin. Neurophysiol.,vol. 120, no. 1, pp. 24–29, Jan. 2009.

[39] Y.-H. Liu, S. Huang, H.-C. Huang, and W.-H. Peng, “Novel motorimagery-based brain switch for patients with amyotrophic lateral scle-rosis: A case study using two-channel electroencephalography,” IEEEConsum. Electron. Mag., vol. 8, no. 2, pp. 72–77, Mar. 2019.

[40] R. Xu et al., “Continuous 2D control via state-machine triggered byendogenous sensory discrimination and a fast brain switch,” J. NeuralEng., vol. 16, no. 5, Jul. 2019, Art. no. 056001.

[41] G. Pfurtscheller, “The hybrid BCI,” Frontiers Neurosci., vol. 4, p. 3,Apr. 2010.

[42] Y. Tomita, F.-B. Vialatte, G. Dreyfus, Y. Mitsukura, H. Bakardjian, andA. Cichocki, “Bimodal BCI using simultaneously NIRS and EEG,” IEEETrans. Biomed. Eng., vol. 61, no. 4, pp. 1274–1284, Apr. 2014.

[43] B. Koo et al., “A hybrid NIRS-EEG system for self-paced braincomputer interface with online motor imagery,” J. Neurosci. Methods,vol. 244, pp. 26–32, Apr. 2015.

[44] Y.-J. Kim and E. C. Lee, “EEG based comparative measurement ofvisual fatigue caused by 2D and 3D displays,” in Proc. Int. Conf. Hum.-Comput. Interact., 2011, pp. 289–292.

[45] M. J. Khan and K.-S. Hong, “Passive BCI based on drowsiness detection:An fNIRS study,” Biomed. Opt. Express, vol. 6, no. 10, pp. 4063–4078,Oct. 2015.

[46] A. Myrden and T. Chau, “Effects of user mental state on EEG-BCIperformance,” Frontiers Hum. Neurosci., vol. 9, p. 308, Jun. 2015.

[47] C.-H. Han, Y.-W. Kim, D. Y. Kim, S. H. Kim, Z. Nenadic, andC.-H. Im, “Electroencephalography-based endogenous brain–computerinterface for online communication with a completely locked-in patient,”J. NeuroEng. Rehabil., vol. 16, no. 1, p. 18, Dec. 2019.

[48] J. Shin, D.-W. Kim, K.-R. Müller, and H.-J. Hwang, “Improvementof information transfer rates using a hybrid EEG-NIRS brain-computerinterface with a short trial length: Offline and pseudo-online analyses,”Sensors, vol. 18, no. 6, p. 1827, Jun. 2018.

[49] J. Shin, K.-R. Müller, C. H. Schmitz, D.-W. Kim, and H.-J. Hwang,“Evaluation of a compact hybrid brain-computer interface system,”BioMed Res. Int., vol. 2017, pp. 1–11, Mar. 2017.

[50] J. Shin, A. von Lühmann, D.-W. Kim, J. Mehnert, H.-J. Hwang, andK.-R. Müller, “Simultaneous acquisition of EEG and NIRS duringcognitive tasks for an open access dataset,” Sci. Data, vol. 5, no. 1,Dec. 2018, Art. no. 180003.

[51] S. Fazli et al., “Enhanced performance by a hybrid NIRS–EEG braincomputer interface,” NeuroImage, vol. 59, no. 1, pp. 519–529, Jan. 2012.

[52] G. Muller-Putz et al., “Towards noninvasive hybrid brain–computerinterfaces: Framework, practice, clinical application, and beyond,” Proc.IEEE, vol. 103, no. 6, pp. 926–943, Jun. 2015.



[53] J. Shin, J. Kwon, and C.-H. Im, “A ternary hybrid EEG-NIRS brain-computer interface for the classification of brain activation patternsduring mental arithmetic, motor imagery, and idle state,” FrontiersNeuroinf., vol. 12, p. 5, Feb. 2018.

[54] S. Fazli, S. Dähne, W. Samek, F. Bießmann, and K.-R. Müller, “Learningfrom more than one data source: Data fusion techniques for sensorimotorrhythm-based brain–computer interfaces,” Proc. IEEE, vol. 103, no. 6,pp. 891–906, Jun. 2015.

[55] I. Winkler, S. Haufe, and M. Tangermann, “Automatic classification ofartifactual ICA-components for artifact removal in EEG signals,” Behav.Brain Functions, vol. 7, no. 1, p. 30, 2011.

[56] I. Winkler, S. Brandl, F. Horn, E. Waldburger, C. Allefeld, andM. Tangermann, “Robust artifactual independent component classifica-tion for BCI practitioners,” J. Neural Eng., vol. 11, no. 3, Jun. 2014,Art. no. 035013.

[57] P. T. Lin, K. Sharma, T. Holroyd, H. Battapady, D.-Y. Fei, andO. Bai, “A high performance MEG based BCI using single trial detec-tion of human movement intention,” in Functional Brain Mappingand the Endeavor to Understand the Working Brain. Rijeka, Croatia:IntechOpen, 2013.

[58] M. Gärtner, S. Grimm, and M. Bajbouj, “Frontal midline theta oscil-lations during mental arithmetic: Effects of stress,” Frontiers Behav.Neurosci., vol. 9, p. 96, Apr. 2015.

[59] A. Gevins et al., “Electroencephalogram correlates of highercortical functions,” Science, vol. 203, no. 4381, pp. 665–668,Feb. 1979.

[60] Z. Garakh et al., “EEG correlates of a mental arithmetic task in patientswith first episode schizophrenia and schizoaffective disorder,” Clin.Neurophysiol., vol. 126, no. 11, pp. 2090–2098, Nov. 2015.

[61] G. Bauernfeind, R. Leeb, S. C. Wriessnegger, and G. Pfurtscheller,“Development, set-up and first results for a one-channel near-infrared spectroscopy system/entwicklung, aufbau und vorläufige ergeb-nisse eines einkanal-nahinfrarot-spektroskopie-systems,” Biomedizinis-che Technik/Biomedical Eng., vol. 53, no. 1, pp. 36–43, Feb. 2008.

[62] S. Moghimi, A. Kushki, S. Power, A. M. Guerguerian, and T. Chau,“Automatic detection of a prefrontal cortical response to emotionallyrated music using multi-channel near-infrared spectroscopy,” J. NeuralEng., vol. 9, no. 2, Apr. 2012, Art. no. 026022.

[63] A. V. Medvedev, “Does the resting state connectivity have hemisphericasymmetry? A near-infrared spectroscopy study,” NeuroImage, vol. 85,pp. 400–407, Jan. 2014.

[64] C.-H. Han, H.-J. Hwang, J.-H. Lim, and C.-H. Im, “Assessment ofuser voluntary engagement during neurorehabilitation using functionalnear-infrared spectroscopy: A preliminary study,” J. NeuroEngineeringRehabil., vol. 15, no. 1, p. 27, Dec. 2018.

[65] S. D. Power, T. H. Falk, and T. Chau, “Classification of prefrontal activ-ity due to mental arithmetic and music imagery using hidden Markovmodels and frequency domain near-infrared spectroscopy,” J. NeuralEng., vol. 7, no. 2, Apr. 2010, Art. no. 026002.

[66] G. Pfurtscheller, G. Bauernfeind, S. C. Wriessnegger, and C. Neuper,“Focal frontal (de)oxyhemoglobin responses during simple arithmetic,”Int. J. Psychophysiol., vol. 76, no. 3, pp. 186–192, Jun. 2010.

[67] G. Bauernfeind, R. Scherer, G. Pfurtscheller, and C. Neuper, “Single-trialclassification of antagonistic oxyhemoglobin responses during mentalarithmetic,” Med. Biol. Eng. Comput., vol. 49, no. 9, p. 979, 2011.

[68] B. Blankertz et al., “The berlin brain-computer interface: Progressbeyond communication and control,” Frontiers Neurosci., vol. 10,p. 530, Nov. 2016.

[69] B. Blankertz, R. Tomioka, S. Lemm, M. Kawanabe, and K.-R. Muller, “Optimizing spatial filters for robust EEG single-trialanalysis,” IEEE Signal Process. Mag., vol. 25, no. 1, pp. 41–56,Dec. 2008.

[70] B. Blankertz et al., “Neurophysiological predictor of SMR-based BCIperformance,” NeuroImage, vol. 51, no. 4, pp. 1303–1309, Jul. 2010.

[71] L. Holper and M. Wolf, “Single-trial classification of motor imagerydiffering in task complexity: A functional near-infrared spectroscopystudy,” J. NeuroEngineering Rehabil., vol. 8, no. 1, p. 34, 2011.

[72] K. Tai and T. Chau, “Single-trial classification of NIRS signals duringemotional induction tasks: Towards a corporeal machine interface,”J. NeuroEngineering Rehabil., vol. 6, no. 1, p. 39, Dec. 2009.

[73] J. H. Friedman, “Regularized discriminant analysis,” J. Amer. Statist.Assoc., vol. 84, no. 405, pp. 165–175, 1989.

[74] B. A. S. Hasan and J. Q. Gan, “Unsupervised movement onset detectionfrom EEG recorded during self-paced real hand movement,” Med. Biol.Eng. Comput., vol. 48, no. 3, pp. 245–253, Mar. 2010.

[75] H. Zhang, C. Guan, and C. Wang, “Asynchronous P300-based brain-computer interfaces: A computational approach with statistical models,”IEEE Trans. Biomed. Eng., vol. 55, no. 6, pp. 1754–1763, Jun. 2008.

[76] A. Barachant, S. Bonnet, M. Congedo, and C. Jutten, “A brain-switchusing Riemannian geometry,” in Proc. 5th Int. Brain-Comput. InterfaceConf. (BCI), Graz, Austria, Sep. 2011, pp. 64–67.

[77] S. S. Shapiro and R. S. Francia, “An approximate analysis of variancetest for normality,” J. Amer. Stat. Assoc., vol. 67, no. 337, pp. 215–216,Mar. 1972.

[78] M. B. Brown and A. B. Forsythe, “Robust tests for the equalityof variances,” J. Amer. Stat. Assoc., vol. 69, no. 346, pp. 364–367,Jun. 1974.

[79] M. Ordikhani-Seyedlar, M. A. Lebedev, H. B. D. Sorensen, andS. Puthusserypady, “Neurofeedback therapy for enhancing visual atten-tion: State-of-the-art and challenges,” Frontiers Neurosci., vol. 10,p. 352, Aug. 2016.

[80] N. Peng et al., “Control of a nursing bed based on a hybrid brain-computer interface,” in Proc. 38th Annu. Int. Conf. IEEE Eng. Med.Biol. Soc. (EMBC), Aug. 2016, pp. 1556–1559.

[81] A. P. Buccino, H. O. Keles, and A. Omurtag, “Hybrid EEG-fNIRSasynchronous brain-computer interface for multiple motor tasks,” PLoSONE, vol. 11, no. 1, Jan. 2016, Art. no. e0146610.

[82] R. T. Schirrmeister et al., “Deep learning with convolutional neuralnetworks for EEG decoding and visualization,” Hum. Brain Mapping,vol. 38, no. 11, pp. 5391–5420, Nov. 2017.

[83] A. M. Chiarelli, P. Croce, A. Merla, and F. Zappasodi, “Deep learningfor hybrid EEG-fNIRS brain–computer interface: Application to motorimagery classification,” J. Neural Eng., vol. 15, no. 3, Jun. 2018,Art. no. 036028.

[84] I. Sturm, S. Lapuschkin, W. Samek, and K.-R. Müller, “Interpretabledeep neural networks for single-trial EEG classification,” J. Neurosci.Methods, vol. 274, pp. 141–145, Dec. 2016.

[85] M. Schreuder, J. Höhne, B. Blankertz, S. Haufe, T. Dickhaus,and M. Tangermann, “Optimizing event-related potential based brain–computer interfaces: A systematic evaluation of dynamic stoppingmethods,” J. Neural Eng., vol. 10, no. 3, Jun. 2013, Art. no. 036025.

[86] T. Solis-Escalante, G. Müller-Putz, C. Brunner, V. Kaiser, andG. Pfurtscheller, “Analysis of sensorimotor rhythms for the implemen-tation of a brain switch for healthy subjects,” Biomed. Signal Process.Control, vol. 5, no. 1, pp. 15–20, Jan. 2010.

[87] G. Mueller-Putz, R. Scherer, G. Pfurtscheller, and C. Neuper, “Temporalcoding of brain patterns for direct limb control in humans,” FrontiersNeurosci., vol. 4, p. 34, Jun. 2010.

[88] A. M. Savić, N. M. Malešević, and M. B. Popović, “Feasibility ofa hybrid brain-computer interface for advanced functional electricaltherapy,” Sci. World J., vol. 2014, pp. 1–11, Jan. 2014.

[89] M. Sugiyama, M. Krauledat, and K.-R. Müller, “Covariate shift adap-tation by importance weighted cross validation,” J. Mach. Learn. Res.,vol. 8, pp. 985–1005, May 2007.


/ColorImageDict > /JPEG2000ColorACSImageDict > /JPEG2000ColorImageDict > /AntiAliasGrayImages false /CropGrayImages true /GrayImageMinResolution 150 /GrayImageMinResolutionPolicy /OK /DownsampleGrayImages true /GrayImageDownsampleType /Bicubic /GrayImageResolution 600 /GrayImageDepth -1 /GrayImageMinDownsampleDepth 2 /GrayImageDownsampleThreshold 1.50000 /EncodeGrayImages true /GrayImageFilter /DCTEncode /AutoFilterGrayImages true /GrayImageAutoFilterStrategy /JPEG /GrayACSImageDict > /GrayImageDict > /JPEG2000GrayACSImageDict > /JPEG2000GrayImageDict > /AntiAliasMonoImages false /CropMonoImages true /MonoImageMinResolution 300 /MonoImageMinResolutionPolicy /OK /DownsampleMonoImages true /MonoImageDownsampleType /Bicubic /MonoImageResolution 900 /MonoImageDepth -1 /MonoImageDownsampleThreshold 1.33333 /EncodeMonoImages true /MonoImageFilter /CCITTFaxEncode /MonoImageDict > /AllowPSXObjects false /CheckCompliance [ /None ] /PDFX1aCheck false /PDFX3Check false /PDFXCompliantPDFOnly false /PDFXNoTrimBoxError true /PDFXTrimBoxToMediaBoxOffset [ 0.00000 0.00000 0.00000 0.00000 ] /PDFXSetBleedBoxToMediaBox true /PDFXBleedBoxToTrimBoxOffset [ 0.00000 0.00000 0.00000 0.00000 ] /PDFXOutputIntentProfile (None) /PDFXOutputConditionIdentifier () /PDFXOutputCondition () /PDFXRegistryName () /PDFXTrapped /Unknown

/CreateJDFFile false /Description >>> setdistillerparams> setpagedevice

Documents

Enhanced Performance of a Brain Switch by Simultaneous Use of …ineuro.korea.ac.kr/theme/community/Journal paper/47.pdf · 2020. 10. 12. · such as ﬂickering visual stimuli or