A Novel Approach to Driving Fatigue Detection Using Forehead EOG

Yu-Fei Zhang, Xiang-Yu Gao, Jia-Yi Zhu, Wei-Long Zheng and Bao-Liang Lu* Senior Member, IEEE

Abstract— Various studies have shown that the traditionalelectrooculograms (EOGs) are effective for driving fatiguedetection. However, the electrode placement of the traditionalEOG recording method is around eyes, which may disturbthe subjects’ activities, and is not convenient for practicalapplications. To deal with this problem, we propose a novelelectrode placement on forehead and present an effectivemethod to extract horizon electrooculogram (HEO) and verticalelectrooculogram (VEO) from forehead EOG. The correlationcoefficients between the extracted HEO and VEO and thecorresponding traditional HEO and VEO are 0.86 and 0.78,respectively. To alleviate the inconvenience of manually labellingfatigue states, we use the videos recorded by eye tracking glassesto calculate the percentage of eye closure over time, which is aconventional indicator of driving fatigue. We use support vectormachine (SVM) for regression analysis and get a rather highprediction correlation coefficient of 0.88 on average.

I. INTRODUCTION

Physiological signals have been applied to detect drivingfatigue in the past decades. Among these signals, electroen-cephalography (EEG) and EOG are two promising measure-ments of fatigue [1]. In recent studies, EEG-based methodswere extensively applied to driving fatigue detection [2] [3].In comparison with that of EEG, the amplitude of EOG issignificantly higher, which makes EOG more robust to noisethan EEG. Two critical issues of EOG-based methods fordriving fatigue detection have been investigated recently: oneis the electrode placement to record EOG, and the other ishow to conveniently label fatigue states. This paper proposesnovel approaches to dealing with these two problems.

In traditional EOG-based experiments, EOG signals arerecorded by two pair of electrodes on opposite sides of eyes[4]. It is difficult to cover these positions with a wearabledevice. Therefore, to develop a new electrode placementfor recording EOG signals is important for improving theuser experience during experiment and practical applications.In this study, we present a novel electrode placement onforehead. For fully making use of existing powerful methodsto extract features from traditional EOG, we introduce aneffective approach to extracting forehead HEO and foreheadVEO.

This work was partially supported by the National Basic ResearchProgram of China (Grant No.2013CB329401), the National Natural ScienceFoundation of China (Grant No.61272248), and the Science and TechnologyCommission of Shanghai Municipality (Grant No.13511500200).

Y.-F. Zhang, X.-Y. Gao, J.-Y. Zhu, W.-L. Zheng and B.-L. Lu are withthe Center for Brain-Like Computing and Machine Intelligence, Departmentof Computer Science and Engineering, and the Key Laboratory of ShanghaiEducation Commission for Intelligent Interaction and Cognitive Engineer-ing, Shanghai Jiao Tong University. Corresponding author: Bao-Liang Lu(bllu@sjtu.edu.cn)

In driving simulation experiments of previous studies, thedriving fatigue state is usually provided by the subjects’ self-report or by manually labelling according to the recordedvideos, which is inconvenient for a long time experiment.In this study, we introduce a novel method for automaticallylabelling the fatigue state during the driving process withSMI eye tracking glasses. The percentage of eye closure overtime (PERCLOS) has been proved to be a good indicatorof driving fatigue [5]. The existing methods for calculatingPERCLOS are not robust in practical applications due to thecomplexity of eye detection. The eye tracking glasses is awearable device and can directly record the state of eyeswithout extra work on eye detection, which is more efficientthan traditional cameras [6]. PERCLOS is usually calculatedwith a 60 seconds window, which makes it not suitablefor real-time driving fatigue detection. Besides, the portabledevice which can accurately track the eye movements isusually too expensive[7]. Therefore, the PERCLOS dataobtained from eye tracking glasses is only used as labelsin training model stage. In practical application, the well-trained model can be adopted to predict fatigue states.

II. EXPERIMENTS

A. Procedure and Subjects

The whole experiment was performed on a driving sim-ulator, which had a four-lane national highway simulatingthe real situation. Total seven-channel EOG signals wererecorded by the NeuroScan system at a sampling rate of 500Hz, which then were down-sampled to 125 Hz. A bandpassfilter between 0 and 30 Hz was used to eliminate noisesignals during the recoding process. At the same time, eyetracking glasses was used to record the videos of subjects’eyes. The experiment environment is shown in Fig. 1.

Our driving simulation experiments were carried out by 14subjects, including 10 men and 4 women aged around 22. Allthe subjects were healthy and no wound on the forehead andface. All of them had an afternoon nap habit or went to bedearly than 11:30 pm. They were required to keep eye openbefore they were tired and do nothing irrelevant with thedriving task. The mean period of the experiment was about2 hours for the purpose of insuring the subject to fall intoa sleepy state during driving. The time of our experimentscontained two periods, one from 12:30 am to 14:30 pm, andthe other from 9:30 pm to 11:30 pm.

B. Forehead EOG and Fatigue Measurement

The EOG-based human-machine interfaces have been ex-tensively investigated, and there are plenty of methods toextract features from traditional EOG signals. Compared

7th Annual International IEEE EMBS Conference on Neural EngineeringMontpellier, France, 22 - 24 April, 2015

978-1-4673-6389-1/15/$31.00 ©2015 IEEE 707

(a) Driving simulation system (b) A subject wearing eye track-ing glasses

Fig. 1. The driving simulation system and eye tracking glasses used in theexperiment.

with finding new methods to directly extract features fromforehead EOG, it is more valuable to find effective methodsto separate HEO and VEO from forehead EOG, which is amixed signal.

Following our previous work [8], we designed a novelplacement to extract forehead EOG signals. Fig. 2 showsthe electrode positions of both forehead and traditionalEOGs in our experiment. The fourth electrode is a commonelectrode to both forehead and traditional EOGs. The refer-ence electrode is placed at the left mastoid and the groundelectrode is placed at the right mastoid. Number five andnumber six electrodes are respectively placed at the edge offorehead, and at the same height with number four electrode.Number seven electrode is placed 3cm over the number fourelectrode.

The PERCLOS, which is calculated from the infra-redvideos recorded by eye tracking glasses, was chosen to bethe indicator of driving fatigue in this study.

12

3

4 56

7

Forehead EOG

Traditional EOG

Fig. 2. Electrode placement for traditional and forehead EOG recording

III. METHODS

A. Forehead HEO and VEO Extraction

By extracting HEO and VEO signals from different com-binations of signals recorded by forehead electrodes, wefound an effective method to extract them from foreheadEOG. The correlation coefficients between traditional EOGand extracted EOG were used to evaluate the extractionmethod. The EOG signals recorded by the fourth and seventhelectrodes were used to extract VEO by FASTICA [9]. Thetwo separated signals obtained by FASTICA contained thesignal with high similarity to VEO, and it appeared in thefirst channel.

When the same method was performed on the signalsrecorded by fifth and sixth electrodes to extract HEO, itgot a result of nearly the same correlation with direct

subtraction of them. The channel of the extracted HEO inthese two separated signals did not appear so fixed. Themost similar signal sometimes showed a negative correlationwith traditional HEO. Therefore, a simple but more robustsubtraction method was finally chosen to acquire foreheadHEO. After the EOG being extracted, a median filter wasused to remove the noise. Fig. 3 shows the comparison ofa segment from one subject between traditional EOG andforehead EOG. The upper part is the traditional EOG, whilethe lower part is forehead EOG. From Fig. 3, we can seethat the amplitude of forehead VEO extracted by FASTICAis significant lower than that of traditional VEO.

0 1 2 3 4 5 6 7 8−400

−200

0

200

400

time(s)

EO

G (u

V)

0 1 2 3 4 5 6 7 8−200

−100

0

100

200

300

time(s)

EO

G (u

V)

0 1 2 3 4 5 6 7 8−400

−200

0

200

400

600

time(s)

0 1 2 3 4 5 6 7 8−2

0

2

4

6

8

time(s)

HEO VEO

Fig. 3. Comparison of a segment between traditional EOG and foreheadEOG

B. Feature Extraction and Smoothing

The wavelet methods have been used in EOG featuresextraction [4], [10]. As the wavelet transform is sensitiveto singularity, it obtains a better result than the derivativemethod in detecting blink and saccade. We computed thecontinuous wavelet coefficients at scale 8 with a Mexicanmother wavelet. Based on the previous studies, we introducedthe peak detection algorithm provided by MATLAB toimprove the performance of former threshold-based detectionalgorithm [4]. The peak detection algorithm was performedon the wavelet coefficients to detect the blink and saccade.

Fig. 4 shows a segment of saccade detection. The greenpoints are the positive peaks, and the blue points are thenegative peaks obtained by peak detection algorithm. Thepeaks with a low threshold amplitude have been ignored.We coded the negative and positive peaks into sequences,in which the positive peak was coded into ‘1’, and thenegative was coded into ‘0’. The segment with a ‘01’ or ‘10’subsequence was chosen as a candidate saccade segment.We can get a straight line from the start point to the endpoint of the candidate segment. We calculated the slopevalue of the line and the correlation between the line and thecorresponding segment. When the absolute value of the slope

708

and the correlation were beyond a threshold, the candidatesegment was judged as a saccade segment. The same methodwas performed on blink detection. The main differencebetween the blink detection with the saccade detection is thatthe blink detection check the subsequence ‘010’ and has athreshold of the maximal segment length.

0 0.5 1 1.5 2 2.5 3 3.5 4−400

−200

0

200

400

600

time(s)

Hor

izon

EO

G v

alue

Horizon EOGMexicanCWT

Fig. 4. The saccade detection segment

To reduce the complexity of extracting features from EOG,we detected the blink from extracted VEO and the saccadefrom extracted HEO. After blink, saccade and fixation (blinkor saccade duration) being detected, we calculated the mean,maximum, variance, and derivative of the blink, saccade, andfixation. The features were extracted from a fixed 8 secondsnon-overlap window. We eliminated the features if theircorrelation coefficients with PERCLOS were under a certainthreshold. Table. I shows the finally chosen 36 features.

TABLE ITHE FEATURES EXTRACTED FROM EOG

Group Features

blink mean/maximum of blink rate variance/amplitude variancemaximum/minimum/mean of blink amplitudepower/mean power of blink amplitudemaximum/mean/sum of blink rateblink number

saccade maximum/minimum/mean of saccade rate/saccade amplitudemaximum/mean of saccade rate variance/saccade amplitude variancepower/mean power of saccade amplitudesaccade number

fixation mean/maximum of blink duration variance/saccade duration variancemaximum/minimum/mean of blink duration/saccade duration

A simple but effective moving average filter was used tosmooth the features, which could remove the components ofsmall correlation with fatigue at the same time.

C. PERCLOS Calculating

We used the contour detection algorithm provided byOPENCV to detect the eye closure state (see [6] for detail).After the eye state being detected, we calculated the PERC-LOS with a 60 seconds length window, and then smoothedthe result with a 30 seconds moving average window. Eachexperiment contained 110 minutes PERCLOS data with thefirst 100 seconds and the tail segment being discarded, andthe same method was also performed on the extracted EOGsignals.

D. Data Partition

The EOG data is time dependent, and cannot be randomlydivided. The time for different subjects to fall asleep isdifferent, so the whole data cannot be split with a fixedinterval. The training segment should include both sleepyand awake data. Therefore, we divided the EOG data intoseveral continuous segments in this experiment. To get along segment of fatigue period to predict, the EOG datawas divided into five continuous segments with 22 minuteslength. Among them, the four segments were used to trainmodel, and the rest was used in prediction.

E. Regression Model

After EOG features being extracted, SVM with linearkernel and linear regression model were used to regress thefeatures to PERCLOS data on each subject.

IV. RESULTS AND DISCUSSION

A. Forehead HEO and VEO Extraction

TABLE IIEXTRACTION RESULTS

Subject HEO VEO Subject HEO VEO

1 0.9208 0.9159 8 0.9290 0.95082 0.8794 0.4380 9 0.9361 0.87723 0.8446 0.9432 10 0.8022 0.94644 0.9283 0.5640 11 0.7525 0.57875 0.8830 0.9394 12 0.6905 0.69836 0.8388 0.8221 13 0.8337 0.49487 0.8778 0.8509 14 0.8683 0.9196

Mean±std 0.8561±0.0707(HEO), 0.7814±0.1871(VEO)

The traditional EOG contains HEO and VEO, so theextracting results contain both forehead HEO and VEO.The correlation coefficients between the forehead EOG andthe traditional EOG on different subjects are illustrated inTable II. We also make comparison between two typesof VEO: one is obtained by FASTICA and the other bythe subtraction method. The results are shown in Fig. 5.The ‘VEO-M’ represents the forehead VEO obtained bysubtraction method. The mean correlation coefficients of bothHEO and VEO are 0.86 and 0.78, while VEO obtainedby subtraction method is 0.66. The results show that thecorrelation coefficient obtained by FASTICA is higher thanthe subtraction method. Therefore, in this study, FASTICAis finally chosen to extract VEO.

Using the electrodes on forehead to record the EOGsignals can reduce much discomfort to the subject, especiallywho wear sunglasses or eye-short glasses. A suitable elec-trode placement and the corresponding separating methodneed further research, which is useful for integrating theEOG electrodes into wearable devices.

B. Data Partition

Because different subjects may have different fatigue statesduring the driving simulation, the divided segment to predictis also somewhat different. In our experiment, the fatiguealways happened between the second and fourth segments.

709

1 2 3 4 5 6 7 8 9 10 11 12 13 140

0.2

0.4

0.6

0.8

1

Subjects

Cor

rela

tion

coef

ficen

tVEO VEO−M

Fig. 5. The results of two methods to extract VEO

TABLE IIIDATA SEGMENT PARTITION

#Predicting segment #Subjects

2 2 3 4 5 7 10 113 1 134 6 8 9 12 14

Table III shows the number of the predicting segment foreach subject. The rest four segments not shown in TableIII are used to train the model. From the segment partitionresults, we can see that the prediction segments are mainlydistributed in second and fourth segment.

C. Regression Prediction

0 2 4 6 8 10 12 14 16 18 20 220

0.2

0.4

0.6

0.8

1

time(min)

Fatig

ue s

tate

Original PERCLOSPredition

Fig. 6. Comparison between predicting fatigue curve and the originalPERCLOS curve

Once the data were collected, the training segments wereused to train the regression model and then the trained modelwas used to predict the test segment. Fig. 6 shows a subject’sPERCLOS prediction with a segment of 22 minutes length,in which the red curve is the predicting results of the SVMand the black curve is the original PERCLOS curve.

The correlation coefficients of prediction results on all the14 subjects are shown in Table IV. From Table. IV, we cansee that the linear SVM gets a correlation coefficient of 0.88on average, while the linear regression (LR) gets 0.74.

The PERCLOS is an easy accessible indicator to drivingfatigue. With the assistance of eye tracking glasses we canget a more accurate PERCLOS data with less extra work.As eye tracking glasses is expensive, it can only be used to

TABLE IVPREDICTION RESULTS

Subject SVM LR Subject SVM LR

1 0.9829 0.9799 8 0.9512 0.94422 0.7656 0.6335 9 0.8063 0.78823 0.9015 0.7785 10 0.8812 0.85004 0.9818 0.9898 11 0.8059 0.63355 0.8786 0.5740 12 0.9309 0.42156 0.9772 0.4215 13 0.8924 0.85507 0.9263 0.9349 14 0.6665 0.6233

Mean±std 0.8820±0.0921(SVM), 0.7448±0.1952(LR)

get the labels in training stage. After the model being trainedwell, the information extracted from forehead EOG signalsis used to predict the fatigue state.

V. CONCLUSION

In this study, we have introduced a novel electrode place-ment on forehead and proposed a corresponding algorithm toextract HEO and VEO from forehead EOG. The extractionresults on 14 subjects show that the separating algorithmworked well on the proposed forehead electrode placement.The forehead is a promising placement to record EOG, whichis more convenient for practical applications. The PERCLOScalculated from the videos recorded by eye tracking glassesprovides a reliable index of driving fatigue. The regressionmodel trained on forehead EOG achieves a fine predictionperformance on each subject. The experiment is a meaningfultrial for using forehead EOG to detect driving fatigue infuture practical applications.

REFERENCES

[1] S. K. Lal and A. Craig, “A critical review of the psychophysiology ofdriver fatigue,” Biol Psychol, vol. 55, no. 3, pp. 173–194, 2001.

[2] C.-H. Chuang, Y.-P. Lin, L.-W. Ko, T.-P. Jung, and C.-T. Lin,“Automatic design for independent component analysis based brain-computer interfacing,” in Proc. 35th Ann. Int. Conf. IEEE EMBS, pp.2180–2183, 2013.

[3] L.-C. Shi and B.-L. Lu, “EEG-based vigilance estimation usingextreme learning machines,” Neurocomputing, vol. 102, pp. 135–143,2013.

[4] A. Bulling, J. A. Ward, H. Gellersen, and G. Troster, “Eye move-ment analysis for activity recognition using electrooculography,” IEEETrans. Patt. Anal. Mach. Intell, vol. 33, no. 4, pp. 741–753, 2011.

[5] W. W. Wierwille, S. Wreggit, C. Kirn, L. Ellsworth, and R. J.Fairbanks, “Research on vehicle-based driver status/performance mon-itoring; development, validation, and refinement of algorithms fordetection of driver drowsiness. final report,” Tech. Rep., 1994.

[6] X.-Y. Gao, Y.-F. Zhang, W.-L. Zheng, and B.-L. Lu, “Evaluatingdriving fatigue detection algorithms using eye tracking glasses,” inProc. 7th Int. IEEE/EMBS Conf. Neural Eng, 2015.

[7] A. Picot, A. Caplier, and S. Charbonnier, “Comparison between EOGand high frame rate camera for drowsiness detection,” in Proc. IEEEWorkshop Appl. Comput. Vis, pp. 1–6, 2009.

[8] H.-Y. Cai, J.-X. Ma, L.-C. Shi, and B.-L. Lu, “A novel method forEOG features extraction from the forehead,” in Proc. 33th Ann. Int.Conf. IEEE EMBS, pp. 3075–3078, 2011.

[9] A. Delorme and S. Makeig, “EEGLAB: an open source toolbox foranalysis of single-trial EEG dynamics including independent compo-nent analysis,” J. Neurosci Methods, vol. 134, no. 1, pp. 9–21, 2004.

[10] Y. Jiao, Y. Peng, B.-L. Lu, X. Chen, S. Chen, and C. Wang, “Recog-nizing slow eye movement for driver fatigue detection with machinelearning approach,” in Proc. Int’l Joint Conf. Neural Networks, pp.4035–4041, 2014.

710