REAL-TIME DRIVER FATIGUE OR DROWSINESS DETECTION … · Real-Time Driver Fatigue or Drowsiness Detection System Using Face Image Stream 795 [email protected] 3

http://www.iaeme.com/IJCIET/index.asp 793 [email protected]

International Journal of Civil Engineering and Technology (IJCIET) Volume 8, Issue 11, November 2017, pp. 793–802, Article ID: IJCIET_08_11_081

Available online at http://http://www.iaeme.com/ijciet/issues.asp?JType=IJCIET&VType=8&IType=11

ISSN Print: 0976-6308 and ISSN Online: 0976-6316

© IAEME Publication Scopus Indexed

REAL-TIME DRIVER FATIGUE OR

DROWSINESS DETECTION SYSTEM USING

FACE IMAGE STREAM

Ilakiyaselvan N, Thomas Abraham J V, Muralidhar A, Kalluri Vinay Reddy

School of Computing Science and Engineering, VIT, Chennai - 600127 Tamilnadu, India

ABSTRACT

According to the Road Accident Statistics in India report presented by the NDTV

in 2016. One death for every 4 minutes because of a road accident in India. The major

reasons for the following accidents are happening due to the Drowsiness and driver

fatigue nature during the driving. Lack of sleep is one of the major reasons for

drowsiness. The boundary between feeling sleepy and established sleep is known as

drowsiness. Driver Fatigue is one major factor for 20% global accidents. There are

many system attempted to identify the drowsiness of the driver in a driving simulation

by recording the Non-visual signals and some systems concentrated on the developing

a face recognition technique while driving and alerting the driver by rising the alarm

in computer software application based interface. This research is aimed to build a

real-time application which can be used to detect the fatigue or drowsiness in driving

conditions and alert the driver whenever drowsiness is detected. The application is

able Predict the behavior of driver by measuring the visual measurements in the

driving conditions in a feasible model which can be used by the all types of

commercial and personal vehicles.

Keywords: Drowsiness, Driver fatigue, Face recognition

Cite this Article: Ilakiyaselvan N, Thomas Abraham J V, Muralidhar A, Kalluri

Vinay Reddy, Real-Time Driver Fatigue or Drowsiness Detection System Using Face

Image Stream, International Journal of Civil Engineering and Technology, 8(11),

2017, pp. 793–802

http://www.iaeme.com/IJCIET/issues.asp?JType=IJCIET&VType=8&IType=11

1. INTRODUCTION

In study, it is found that despite of developing the heavy-end safety measures on the road

development. Accidents are still happening in a large frame across the world. The major

numbers of accidents are due to the drowsiness of the driver [6]. In order to reduce the

accidents a real-time monitoring system has to develop to detect the visual features of the

driver which will measure the fatigue or drowsiness during the driving environment [7]. It is

found that if a driver going in a speed of 100 Km/h and falls asleep for just four seconds

without the control of the driver the vehicle is going to travel a distance of 111 meters in a

Ilakiyaselvan N, Thomas Abraham J V, Muralidhar A and Kalluri Vinay Reddy


Highway [8]. At that speed and distance travelled results to a likely crash. The drivers who

are falling under this category are young drivers, Shift workers in industry which includes the

heavy vehicle drivers and the drivers with the sleep disorders. Study conducted by the

Adelaide Centre for sleep research put some interesting facts regarding drowsiness [8]. They

are like a person who has been awake for the 17 hours faces the same risk of a crash as a

person who has a Blood Alcohol Content reading of 0.05/100ml. They are therefore twice as

likely to have an accident as a person with Zero blood alcohol content who is not fatigues.

The present technology in detection of drowsiness have been classified into four types: [9].

• Measuring Visual Features like eyes, head movement, yawning.

• Measuring Non-Visual Features like EEG and ECG.

• Vehicle position in Lane Monitoring and Steering Pattern Monitoring.

The systems that is dependable on body sensors for measure parameters like brain

activity, heart rate, skin conductance and muscle activity which are difficult to the drivers to

have access to them which causes little uncomforting during driving making them

contactable. The aim of the research that has been taken here is to understand what the real

conditions of driving needs to be followed and what are the expectations from the driver while

using the application[10] [11]. Developing an Android based application which can be placed

in the vehicle without distracting the user can help the driver to come out of the fatigue or

drowsiness while driving. The present work is concentrated on the detection of open and

closed state of human eye using Google Vision library which will parallel help to detect the

facial landmarks. Our method is able to detect eye closure and predict with maximum

accuracy even in the presence of face and eye titling and slightly variations in the light

available around the vehicle.

2. RELATED WORK

The development of drowsiness detection system started back in 1990’s. Considering the

accidents that are happening researchers Hiroshi Ueno with the rest of the team started

analyzing the major reasons for the accidents and found fatigue is one such reason. Images are

used to detect the drowsiness. Increase of time to detect eye state and keeping limited

resources of hardware. The system came up with a little reduced Alertness.[12]

Later many researchers have done significant work to detect drowsiness keeping available

technology. One such attempt which showed significant features extraction done by

researcher Antoine picot, Sylvi and Alice Caplier where they worked on complete visual signs

of eyes through high frame rate video. By recorded video analysis they are able to detect the

drowsiness from the videos but accuracy and efficiency has not improved to the expected

value but proposed the basic algorithm to detect the fatigue.[13]

Later in 2012 to present use enhanced algorithms like viola-johns Object detection

Framework[6], openCV[15] and PERCLOS[6], [9], [16] taken major turn in detecting the

drowsiness many researchers have put forward there work based on eye closure detection.

From all the related work study the requirement of the driver has been analyzed has[4]:

• It should be able to fit in any kind of personal and commercial vehicles which includes

heavy trucks.

• The application should not cause any distraction to the driver. The negative and false

alarm rate should be minimum throughout the journey.

• The system should be able to work in real-time. Able to work in both day and night

time.

Real-Time Driver Fatigue or Drowsiness Detection System Using Face Image Stream


3. PROPOSED SOLUTION TO DETECT DROWSINESS AND SYSTEM

ARCHITECTURE

Figure 1 System Architecture Drowsiness Detection System

To build a real-time face and eye tracking application which can detect the driver fatigue

or drowsiness in driving conditions. The application is embedded and contactless to the driver

for the comfort while driving. It is able to detect the drowsiness and alert the user with in a

threshold time.

It is designed in such a way the tilted action from the face will go along with the detection.

This will make driver advantage of moving inside the vehicle. The application is able to

classify the state of the eye during the driving condition. Based on the percentage of eye

closure the drowsiness alert has been classified into three types: Awake Event, Slightly

Drowsy, and Drowsy. The application is built on the android platform. Whenever the eyelids

are closed to more than 80%. The application will raise an alarm and alert the Driver as



shown in figure 1.. The application is built upon Google Mobile Vision Library[17] and used

Google play Services.

4. APPLICATION DEVELOPMENT AND APPROACH

The modules in the project are broadly classified in to four stages of development as:

• Image Capturing

• Face Movement Detection

• Eye Motion Detection

• Drowsiness Detection and Alertness

Each stage will generate on sort of output which will be input to other stage of Module

4.1. Image Capturing

It requires permission from the user to access the front camera. It should be able to swap

width and height sizes when in portrait rotate by 90 degrees during the movement in face.

Camera hardware

The camera should be able to produce at least four frames per second through camera which

will be accessed by the android device. During night time Near-infrared light can be used to

keep it ready mode to capture the image. The light sensitive can be ranging from 750-2000

nm. For this application in order to achieve high accuracy nearly 60 fps powered camera is

being used. The application even have capability to run four frames per second[17].

4.2. Face Movement Detection

The face movement detection is crucial and important to recognize the facial expression of the

driver in driving condition. An overlay has to be formed to confirm the single face in the

frame which is near to the camera. Multiple faces cannot be detected during face movement

detection.

Figure 2 Face Movement detection

Haar like features in Face Movement Detection

These features are generally used to detect the face and eye in the captured image.

Rectangular haar-like feature is the difference between the summation pixels of areas which

are inside the rectangle at any position and scale within the original image. The advantage of



using haar like features over raw pixels values is that it can reduce/increase the in-class/out-

of-class variability, which means the classification.

Application of Haar Classifier

The frames that are obtained from camera which includes area of resolution of 640*480

height and width respectively. The Haar like features are applied to the image which captures

through the application in every location and at every scale. From the face geometry the

region of interest is selected from the face region. Now in this ROI Haar cascade for closed

eyes is applied if it detects a closed eye a counter increments and camera fetches next frame

and it is processed. If closed eye is not detected Haar classifier for open eyes are checked.

After it is complete it takes the next frame from the camera and processes it. Every time the

PERCLOS value is calculated as the ratio between numbers of closed eyes detected and

number of eyes open found. From the obtained threshold of PERCLOS, fatigue level decision

is taken, and can be used to alarm the driver.

Tilted Face Detection with Affine Transformation Matrix

The original Haar cascade technique applied for face detection detects upright face only. If

there is a moderate amount of tilt of face it will not be detected, consequently eyes will also

be not detected in such a frames. Since the angle in which driver faces camera can vary with

the driving conditions, it is necessary to detect the tilted face. In order to detect we have

implemented a method based on affine transformation matrix.

Affine Transformation Matrix

An affine transformation linear 2-D geometric function which can map variables of an input

image into new variables by applying linear combination of translation, rotation. The

advantages of using affine transformation are that it preserves collinearity and ratio of

distance. These two properties assures that the affine transformed faces will be detected by the

Haar classifiers[18]. In 2-D graphics, for rotation by angle θ counter clockwise about the

origin written in matrix form as:

��′� = �� − ��

��

The rotation matrix can be found for an “n” dimensions image once it its size, centre and

angle of rotation needed are known. This is implemented in the algorithm.

Steps used to Detect Tilted Face

• First Haar classifier for the face is applied

• If face is not detected then entire captured images is transformed or rotated to close

wise and counter clockwise.

• After the detection the co-ordinates of face are mapped to original image after de-

rotation.

• The eye detection method gives de-rotated eye region.

• It marks further processing more accurate. These steps improves the angle of tilt for

face detection more than 45 degree, and it returns the ROI for eye detection in a de-

rotated form, it further enhances the eye rotation rate.



Face Tracking

Searching for face in every frame in every scale increases the computational complexity. The

real-time performance of the algorithm can be implemented if we use the temporal

information. If the position and size of face is known accurately in a frame, then we can select

ROI around that position where we can find the face in subsequent frame. The computational

complexity is less since the search region is reduced. Application of KLT algorithm is used

for face tracking[19].

Step 1: Track the Face of the driver

Step 2: Make a copy of the points to be used for computing and locating ROI of the

geometric.

Step 3: Transformation between the points in the previous and the current frames

respectively to the user movement.

Step 4: Get the next Frame in the video sequence.

Video_Frame = Get_Next_Frame (VideoFileReader)

Step 5: Track the points in the ROI. (Need at least 2 points along the axis)

Step 6: Estimate the geometric transformation between the old points and the new points

and eliminate outliers using linear translation. (Minimum Four Frames are required to

calculate).

Step 7: Insert a bounding rectangle box around the object being tracked.

Step 8: Display tracked points.

Video Points = insertMarker (VideoFrame, VisibleParts, ‘+’… ‘Colour’, ‘White’);

Step 9: Reset the points and display the annotated frame using android application.

4.2. Eye Motion Detection.

Once face is localized next step is to detect the position of eye. Subsequently the eye detected

is classified to open or close[15]. The detection of eye in the face region is modeled as an

object detection problem. Haar classifier based eye detection on original obtained camera

frame. One user-defined classifiers for open eye and closed eye are used in this detection

process. The classifier for open and close are trained with a database of positive and negative

images are taken into consideration. The ROI selection is done and the detection of eye is

performed in the localized region[20].

4.3. Detect Drowsiness using PERCLOS Calculation.

The number of open and closed eyes over one minute windows are calculated and PERCLOS

values are found. Different Stages of opening and closing eyes values and variables are taken

into consideration. They are Awake, Drowsy, Eyes Closed, Eyes Opened, Likely drowsy,

Normal eye blink, pending slow eye lid closure, slow eye lid closure events in declaring to

check the PERCLOS value for one minute. Threshold for detecting drowsiness is fixed as:

With Drowsy= 0.8 Sec and With Likely Drowsy= 0.15 Sec

PERCLOS will always compare between the previous event and the actual event.

PERCLOS= (Dm - Da)/ Dm

Dm: It is the frame numbers captured in one minute

Da: It is the frames of the eye belong to attentive category.

Dm - Da: It is the number of eye frame belonging to the inattentive category.



5. WORKING MODEL OF THE SYSTEM

The application is developed using android platform and Google Mobile Vision API[17]. The

below screenshots of the android application which states the different stages of drowsy and

active states that may cause to particular person while driving. Apart from raising the alarm

the usage of colour based alertness is done each and every colour has their own representation

of alertness.

Green colour (Fig.3) indicates the active state of the driver. Yellow colour (Fig.4)

indicates the likely drowsy state of the driver. Red colour (Fig.5) indicates the drowsy nature

beyond threshold of the driver. The application works even in the tilted position while driving

to the extent of 45 degrees so it can be real time even during the driving conditions. Face

tilted has to be detected because the drivers are not going to stay in ideal condition while

driving. There will be a motion in driving and it has to ready to go along with the road

because little vibrations. The up-ward and down-ward movement of face will be detected. If

in-case there some drivers lean down for up while feeling drowsy. The possibilities of feeling

in those directions are more. More chances of accidents occur at that situation.

Figure 3 Active State Figure 4 Likely Drowsy Figure 5 Drowsy State

Figure 6 Right tilt of the face Figure 7 Left tilt of the face



6. FUTURE WORK AND CONCLUSION

The application that developed supports and works in android mobile phones and tablets. In

the current android market place the app is supported in 97.4% android devices. The proposed

and developed application can help to detect the drowsiness and likely drowsiness of the

driver. By analyzing different stages of the opening and closing eyes the drowsiness is

detected. The use of application can detect the drowsiness and alert the driver is sleeping

conditions and alert the driver by considering the speed of the vehicle at 80-100 km/hr the

threshold is fixed and the same used to make the alarm. A lot of the work can be developed

towards detecting the drowsiness using all types of body motion sensors and eye detection

sensor and enabling semi-automatic brakes to the vehicle whenever the driver is about to feel

sleepy. Other issues which can improve the detection system is to provide with good NIR

lighting facility to the mobile devices which can detect the faces in the night time. This real-

time application can definitely make impact in reducing number of accidents occurring and

increase the road safety of the Driver.

REFERENCES

[1] NDTV, Road Accident Statistics in India, 2016. [Online]. Available:

http://sites.ndtv.com/roadsafety/important-feature-to-you-in-your-car-5/.

[2] H. Kang, Various Approaches for Driver and Driving Behavior Monitoring : A Review,

pp. 616–623, 2013.

[3] K. Srijayathi and M. Vedachary, Implementation of the Driver Drowsiness Detection

System, vol. 2, no. 9, pp. 1751–1754, 2013.

[4] Pressreader, Driver fatigue Global Accidents, 2017. [Online]. Available:

https://www.pressreader.com/south-africa/pretoria-news-

weekend/20161217/281535110633688.

[5] C. T. Lin, C. J. Chang, B. S. Lin, S. H. Hung, C. F. Chao, and I. J. Wang, A real-time

wireless brain-computer interface system for drowsiness detection, IEEE Trans. Biomed.

Circuits Syst., vol. 4, no. 4, pp. 214–222, 2010.

[6] T. Pradhan, A. N. Bagaria, and A. Routray, Measurement of PERCLOS using eigen-eyes,

4th Int. Conf. Intell. Hum. Comput. Interact. Adv. Technol. Humanit. IHCI 2012, pp. 0–3,

2012.

[7] L. M. Bergasa, J. M. Buenaposada, J. Nuevo, P. Jimenez, and L. Baumela, Analysing

driver’s attention level using computer vision, IEEE Conf. Intell. Transp. Syst.

Proceedings, ITSC, pp. 1149–1154, 2008.

[8] V. S. Government, Fatigue Statistics. [Online]. Available: http://www.tac.vic.gov.au/road-

safety/statistics/summaries/fatigue-statistics.

[9] J. J. Yan, H. H. Kuo, Y. F. Lin, and T. L. Liao, Real-time driver drowsiness detection

system based on PERCLOS and grayscale image processing, Proc. - 2016 IEEE Int.

Symp. Comput. Consum. Control. IS3C 2016, pp. 243–246, 2016.

[10] S. Begum, Intelligent driver monitoring systems based on physiological sensor signals: A

review, IEEE Conf. Intell. Transp. Syst. Proceedings, ITSC, no. Itsc, pp. 282–289, 2013.

[11] M. Ben Dkhil, A. Wali, and A. M. Alimi, Drowsy Driver Detection by EEG Analysis

Using Fast Fourier Transform, pp. 313–318.

[12] H. Ueno, M. Kaneda, and M. Tsukino, Development of drowsiness detection system,

Proc. VNIS’94 - 1994 Veh. Navig. Inf. Syst. Conf., pp. 15–20, 1994.

[13] A. Picot, S. Charbonnier, and A. Caplier, Drowsiness detection based on visual signs:

blinking analysis based on high frame rate video, 2010 IEEE Instrum. Meas. Technol.

Conf. Proc., pp. 801–804, 2010.



[14] Q. Wu, B. X. Sun, B. Xie, and J. Zhao, A PERCLOS-based driver fatigue recognition

application for smart vehicle space, Proc. - 3rd Int. Symp. Inf. Process. ISIP 2010, pp.

437–441, 2010.

[15] J. F. Xie, M. Xie, and W. Zhu, Driver fatigue detection based on head gesture and

PERCLOS, 2012 Int. Conf. Wavelet Act. Media Technol. Inf. Process. ICWAMTIP 2012,

pp. 128–131, 2012.

[16] G. C. vision R. Team, Google Mobile Vision, 2017. [Online]. Available:

https://developers.google.com/vision/.

[17] S. Darshana, D. Fernando, S. Jayawardena, S. Wickramanayake, and C. De Silva,

Efficient PERCLOS and gaze measurement methodologies to estimate driver attention in

real time, Proc. - Int. Conf. Intell. Syst. Model. Simulation, ISMS, vol. 2015–Septe, pp.

289–294, 2015.

[18] P. R. Tabrizi and R. A. Zoroofi, Open/closed eye analysis for drowsiness detection, 2008

1st Int. Work. Image Process. Theory, Tools Appl. IPTA 2008, 2008.

[19] C. Bila, F. Sivrikaya, M. A. Khan, and S. Albayrak, Vehicles of the Future: A Survey of

Research on Safety Issues, IEEE Trans. Intell. Transp. Syst., vol. PP, no. 99, pp. 1–20,

2016.

[20] T. Hayami, K. Matsunaga, K. Shidoji, and Y. Matsuki, Detecting drowsiness while

driving by measuring eye movement- A pilot study, IEEE Conf. Intell. Transp. Syst.

Proceedings, ITSC, vol. 2002–Janua, no. September, pp. 156–161, 2002.

[21] Suganya G, Premalatha M, Bharathiraja S and Rohan Agrawal, A Low Cost Design to

Detect Drowsiness of Driver, International Journal of Civil Engineering and Technology,

8(9), 2017, pp. 1138–1149

[22] Jyoti Verma, Vineet Richariya, Face Detection and Recognition Model Based on Skin

Colour and Edge Information for Frontal Face Images, International Journal of Computer

Engineering & Technology (IJCET), Volume 3, Issue 3, October - December (2012), pp.

384-393

[23] Walaa M Abdel-Hafiez, Mohamed Heshmat, Moheb Girgis, Seham Elaw, A New Face

Recognition Scheme for Faces with Expressions, Glasses And Rotation, International

Journal of Computer Engineering & Technology (IJCET), Volume 5, Issue 4, April

(2014), pp. 11-23

[24] Ram Ratan Ahirwal, Sagar Nachankar, Y. K. Jain, A Novel Method of Average Filtering

for Removing Noise and Face Recognition, International Journal of Computer

Engineering & Technology (IJCET), Volume 5, Issue 8, August (2014), pp. 55-70

[25] Shashikant Sharma, Kota Solomon Raju, Application of Gaussian Filter with Principal

Component Analysis Algorithm for The Efficient Face Recognition, International Journal

of Electronics and Communication Engineering & Technology (IJECET), Volume 4,

Issue 7 (2013), pp. 244-251

[26] S. K. Hese, M. R. Banwaskar, Appearance Based Face Recognition by Pca and Lda,

International Journal of Electronics and Communication Engineering & Technology

(IJECET), Volume 4, Issue 2, March – April, 2013, pp. 48-57

[27] Archana H. Sable, Dr. Girish V. Chowdhary, A Two Phase Algorithm for Face

Recognition in Frequency Domain, International Journal of Computer Engineering &

Technology (IJCET), Volume 4, Issue 6, November - December (2013), pp. 127-135

[28] Smt. Mahananda D. Malkauthekar, Smt. Shubhangi D. Sapkal, Comparison of

Mahalanobis and Manhattan Distance Measures in Pca Based Face Recognition,

International Journal of Computer Engineering & Technology (IJCET), Volume 5, Issue

5, May (2014), pp. 01-11

[29] Keyur Shah, Vijay Ukani, Efficient Face Recognition System using Hybrid Methodology,

International Journal of Advanced Research In Engineering and Technology (IJARET),

Volume 5, Issue 4, April (2014), pp. 179-189



[30] Abhishek Choubey, Girish D. Bonde, Face Recognition across Pose with Estimation of

Pose Parameters, International Journal of Electronics and Communication Engineering &

Technology (IJECET), Volume 3, Issue 1, January- June (2012), pp. 311-316

[31] Bilal Salih Abed Alhayani, Prof. Milind Rane, Face Recognition System by Image

Processing, International Journal of Electronics and Communication Engineering &

Technology (IJECET), Volume 5, Issue 5, May (2014), pp. 80-90

[32] Mrs. Manisha Bhisekar, Prof. Prajakta Deshmane, Image Retrieval and Face Recognition

Techniques: Literature Survey, International Journal of Electronics and Communication

Engineering & Technology (IJECET), Volume 5, Issue 1, January (2014), pp. 52-58

[33] Prof. B.S PATIL, Prof. A.R YARDI, Real Time Face Recognition System Using Eigen

Faces, International Journal of Electronics and Communication Engineering &

Technology (IJECET), Volume 4, Issue 2, March – April, 2013, pp. 72-79

[34] J. V. Gorabal, Manjaiah D. H, Texture Analysis for Face Recognition, International

Journal of Graphics And Multimedia (IJGM), Volume 4, Issue 2, May - December

2013,pp 20-30

Documents

REAL-TIME DRIVER FATIGUE OR DROWSINESS DETECTION … · Real-Time Driver Fatigue or Drowsiness Detection System Using Face Image Stream 795 [email protected] 3