View
216
Download
0
Category
Preview:
Citation preview
8/9/2019 Seminar 8th Eye Blink
1/20
Blink Detection and Eye Tracking for Eye Localization
1. INTRODUCTION
A method of using frame differencing coupled with optical flow computation for
eye blink detection is proposed. Frame differencing allows quick determination of
possible motion regions. If they are detected, optical flow is computed within these
regions. The direction and magnitude of the flow field are then used to determine
whether a blink has occurred. The eyes are then tracked using the Kanade Lucas
Tomasi (KLT) tracker. We obtained a success rate of 97.0% in blink detection usingthe proposed method, and localised the eyes automatically at an average rate of 22
frames per second.
Eye localization is useful in applications such as face normalization, gaze-based
human-computer interfaces, and security systems using the human iris for identification.
Blink detection also allows measurement of blinking rate and the rate of closures of
eyes. These cues are often used as indicators of driver awareness while operating
vehicles [7], [10].
Methods of blink detection:
A few methods have been proposed for blink detection. In
Yano et al. [11], frame differencing is used for blink
detection. Al-Qayedi and Clark [1] track features about the
eyes and infer blinks through detection of changes in the
eye shape. Smith et al. [8] try to differentiate between
occlusion of the eyes (due to rotation of the head) and
blinking. The subjects sclera is detected using intensity
information to indicate whether the eyes are open or closed
8/9/2019 Seminar 8th Eye Blink
2/20
(i.e., a blink is occurring). Black et al. [4] detect blink using
optical flow but the system restricts motion of the subject
and needs near frontal views in order to be effective. The
reported 65% success rate in detecting blinks seems to be
too low for applications such as communicating via blinks.
Grauman, et al. [5] report a success rate of 96.5% in almost
real-time. Here, frame differencing is initially performed to
obtain motion regions. Then a pre-trained open-eye
template is selected for the computation of the Mahalanobis
distance measure which is used to retain the best pair of
blobs as candidates for eyes. The eyes are tracked and
correlation scores between the actual eye and the
corresponding closed-eye template are used to detect
blinks. However, it requires offline training for different
depths from the camera for the computation of the distance.
In addition, the system requires initialization in which a
stored template for a particular depth is chosen. Once the
template is chosen and the system is in operation, the
subject will be restricted to be at the specified distance.
Another disadvantage of the system is that changing camera
positions requires the whole system to be retrained.
Furthermore, the same system may not be as effective if it
were used on people of different races with disparate eye
sizes and distance between the eyes.
The algorithm proposed here reduces computational costs
by first performing frame differencing to localize motion
regions.
However, the detected regions may arise due
various causes such as eye blinks, eyeball motion, head
8/9/2019 Seminar 8th Eye Blink
3/20
motion and possibly background motion. To differentiate
between blinks and other types of motions, we use optical
flow and the fact that during an involuntary blink, the eyes
are first covered and then uncovered within a window of a
few consecutive few frames (typically 4-5). Our goal is to
create a system which allows
(i) the subject relative freedom of motion
(as long as the eye sizes are between 500 and
5000 pixels and there are no drastic movements in
consecutive frames)
(ii) the operator the flexibility to place
the camera at any suitable location and yet achieve a high
success rate.
2. PROPOSED ALGORITHM
Blink detection allows localization of the eyes, by using
frame differencing followed by optical flow. Subsequently,
the localized eyes are tracked by the Kanade-Tomasi-Lucas
(KLT) [6, 9, 2] method. These are described below.
2.1 Blink Detection
The steps in the blink detection algorithm are describedbelow:
(i) Obtain locations of possible motion using frame
differencing.
8/9/2019 Seminar 8th Eye Blink
4/20
(ii) Suitably threshold the motion regions and obtain blobs
using morphological operations and connected components.
(iii) Remove unsuitable blobs i.e. blobs which are either too
big or too small or have incorrect width to height ratios to
be considered as eyes.
(iv) If no possible candidate blobs are found, repeat (i) to
(iii) on subsequent frames until at least 2 suitable blobs are
found, and mark their positions.
(v) Compute optical flow field in the blob regions.
(vi) Ascertain dominant direction of motion in the blobs. If
the dominant motion is downwards in a pair of blobs, their
positions are noted. These would represent eye closure
during a blink. If the motion is not downwards then steps (i)
to (vi) are repeated. The remaining steps are used for
detecting subsequent eye opening phase of blink, followed
by bounding the eye regions.
(vii) Repeat steps (i) to (iii).
(viii) Discard blobs that are not situated near the location of
the blobs found with downward motion.
(ix) Compute optical flow to ascertain if the dominant
motion is upwards if there are at least 2 blobs remaining,
Otherwise repeat from step (vii) on a new frame.
8/9/2019 Seminar 8th Eye Blink
5/20
(x) If the dominant motion is upwards, then classify the
frames beginning from the frame where downward motion
was detected to the frame where upward motion was
detected as blink frames. If after the downward motion, no
upward motion is detected within the next 3 frames, no
blink is deemed to have occurred (this is because
involuntary blinks in general do not last more than 5
frames) and the process of blink detection starts from step
(i) on new frames.
(xi) The bounding boxes of the blobs where blink is deemed
to have occurred are taken as eye locations.
The computation of optical flow allows for the
differentiation between vertical eyelid movements during
blinks and movement of the eyeball (which is
predominantly horizontal) and horizontal head movements
(Figure 2). Vertical head movements result in blobs that
can be eliminated based on size, ratio and the average
magnitude of the velocity vectors (which is much lower in
the case of head motion. See Figure 3).
2.2Tracking of the Eyes
After having located the eyes, we track them using the KLT
tracker [9], [6], [2]. The tracker is initialized on the frame
immediately after the blink is detected. We found that
twenty feature points gave the best results for tracking
8/9/2019 Seminar 8th Eye Blink
6/20
8/9/2019 Seminar 8th Eye Blink
7/20
accuracy. These twenty feature points are selected in the
area of the eyes where there are edges and then they are
tracked in subsequent frames.
It was found that when there is drastic motion in the eyes
(which can be caused by a blink or motion of the eye ball),
several feature points are lost. If this number is greater than
20% of the total feature points, then the accuracy of the
tracker would be compromised; hence it is disabled and re-
initialized from a newly detected blink frame.
However,
occasionally less than 20% of the feature points were
reported as lost even during a blink. Thus, additional
checks such as height to width ratio of the tracked region
are used to ensure that the tracker will indeed be disabled if
necessary. Reinitialization at every blink enables the tracker
to track the eye accurately even in long sequences.
an eye localizerand face matcher:
Ouranalysis of systems composed of an eye localizerand
face matcher in terms of performance and computational
complexityrevealed that:
1. Worst-case eye localization errors have a big impact
on face matching performance, making eye localizers
that, at identical eye localizer accuracy, have a long-
tail distribution of errors perform worse to ones that
have compactly distributed errors.
8/9/2019 Seminar 8th Eye Blink
8/20
2. Given an eye localizer performance better than our
Priors method and in respect to the previous obser-
vation, eye localization can boost the performance of
naive face matching approaches such as PCA.
3. Given a suitable eye localizer (such as known in liter-
ature), eye localization allows for more ecient face
matching without degrading perfomance. This e-
ciency gain will improve when the number of images
to match increases regardless of the complexity of the
eye localizer.
8/9/2019 Seminar 8th Eye Blink
9/20
3.3 Overview of the visual attention model
The model used in this work [32] employs an algorithm
that assigns high VA scores to pixels where
neighbouring pixel configurations do not match
identical positional arrangements in other randomly
selected neighbourhoods in the image.
This means, for example, that high scores will be associated with
anomalous objects, or edges and boundaries, if those
features do not predominate in the image. A flowchart
describing this process is given in Fig
8/9/2019 Seminar 8th Eye Blink
10/20
Fig Visual attention model.
For display purposes the VA scores for each pixel are
displayed as a map using a continuous spectrum of false
colours with the scores being marked with a distinctive color of gray level.
8/9/2019 Seminar 8th Eye Blink
11/20
2.3 Speed Considerations
To speed up localization and tracking of the eyes, the
parameters for optical flow computations and KLT are
tuned. The time taken for optical flow is the major
component in the time needed for the detection of a blink.
Considerable time is used in the gradient descent iterations
for accurate flow computations. By limiting these to 10
iterations, flow computation required an average of 0.8 sc
per frame; assuming that an average blink lasts 4 frames,
optical flow computations needs 3.2 sc to detect a blink.
For the 13 sequences we tested, the accuracy of blink
detection was unchanged due to this speed modification. As
for increasing the speed of the tracker, the number of points
tracked around the eye region was set to 10. However, the
limitation is that the eyes are not localized as perfectly as
with 20 feature points. The optimal window size about each
point was found to be 11 x 11 in terms of speed and
accuracy.
In our simulations we considered 13 stored video sequences
of 700 frames (768 x 576 PAL frames at 25 fps), obtained
from different people, with each sequence containing an
average of 5 blinks. On a 1.6 GHz Pentium IV computerwe obtained the following average times for each sequence
as below:
8/9/2019 Seminar 8th Eye Blink
12/20
1. Optical flow: 20 secs.
2. Tracking of the eyes: 10 secs.
3. Reading the images from disk: 13 secs.
We postulate if the system is run online, then by avoiding
the disc accesses, a processing rate of about 22 fps can be
obtained.
3. RESULTS ANDDISCUSSION
Figure 1 shows a typical result obtained with optical flow
computation at the start and end of a blink. In our
experiments, we found that optical flow reliably
distinguished between motions such as blinks, sideways
motion of the eyeballs and the head movements. During
activities such as reading and playing games on a computer
monitor, the optical flow vectors arising from vertical head
movements were found to be typically smaller in
magnitude, compared to flow vectors arising from blinks. In
Figures 2 and 3, we see that performing only frame-
differencing can lead to the wrong regions being identified
as eyes. Using only frame differencing fails to distinguish
between motion of the eyeball and a blink as seen in Figure
3. Thus optical flow is necessary to ensure the accuracy of
the localization of the eyes. By comparing the optical flow
results in Figures 1 and 3, it is clear that the magnitude of
the velocity vectors in the case of the blink is much larger
8/9/2019 Seminar 8th Eye Blink
13/20
than in the case of vertical head movement.
( Graph for visualizing blink length/frames )
It was found that the KLT tracker performs well and is able
to track the eyes in the interval between blinks. In our
experiments, we found that for eyes of size about 30x50,
whenever there was a movement of about 9 pixels or more
in the subjects eyeball, the tracker was disabled. This is
undesirable as the tracker should only be disabled by the
8/9/2019 Seminar 8th Eye Blink
14/20
occurrence of a blink. To avoid this, we use the fact that the
dominant direction of the optical flow for eye-ball
movements is horizontal.
If the direction of the flow
vectors is sideways in the motion regions, then eye-ball
movement is deemed to have taken place and the tracker is
reinitialized in the next frame using eye regions from the
previous frame. At every blink, to prevent drifting, the
tracker is reinitialized.
Figure 4 shows the eyes being
reliably tracked in the interval between two blinks.
The algorithm was tested on 13 video sequences of different
people, each of about 700 frames (resolution of the frames
being 768 x 576, captured at 25 fps). Of the 65 blinks in the
sequences two were missed and there were no false
detections, giving an accuracy rate of 97%.
8/9/2019 Seminar 8th Eye Blink
15/20
Figure 1: Results of frame differencing (a,c) and opticalflow computation (b,d). Motion regions obtained by frame
differencing are bounded by black boxes. In (b) majority of
the optical flow vectors from the first 2 frames of a blink,
are pointing downwards. In (d) majority of flow vectors are
pointing upwards in the frames where the eye first reopens.
8/9/2019 Seminar 8th Eye Blink
16/20
Figure 2: (a,b) Motion regions (rectangular regions)
resulting from frame differencing and (c) optical flow
computations when there is horizontal motion of the head
as well as motion of the eye ball. Majority of the optical
flow vectors are pointing sideways.
8/9/2019 Seminar 8th Eye Blink
17/20
Figure 3: (a) Motion regions (rectangles) from frame
differencing and (b) optical flow vectors when there is
vertical head movement. The majority of the flow vectors
are pointing upwards and their magnitude is much smaller
than that seen in Figure 1 (for a blink).
Applications:
Once blinking is detected, measures like blinking rate,
number of eye closures and eye positions are easily
obtained. Possible areas of application for such a system
include gaze-based human-computer interfaces and security
systems.
ACCURACY:
8/9/2019 Seminar 8th Eye Blink
18/20
accuracy rate of 97%
4. CONCLUSIONS
In this paper we have proposed an accurate and fast method
for locating and tracking the eyes of a computer user
situated in front of the monitor. By computing optical flow
and using both the magnitude and direction of the flow
vectors,
we can differentiate blinking from the other
motions. The located eyes are tracked by the KLT tracker.
Based on testing of 13 sequences from different individuals,
a blink detection accuracy of 97% was obtained. We
estimate that the algorithm can run at approximately 22 fps
with accurate tracking.
Once blinking is detected, measures like blinking rate,
number of eye closures and eye positions are easily
obtained. Possible areas of application for such a system
include gaze-based human-computer interfaces and security
systems.
Future work involves development of a faster and robust
optical flow algorithm for this application to reduce the
amount of time required for blink detection. A limitation ofthe proposed scheme is that a blink is necessary to detect
the eyes. However this should not be a problem in practi
8/9/2019 Seminar 8th Eye Blink
19/20
REFERENCES
[1] A. M. Al-Qayedi. and A. F. Clark, Constant-rate eye
tracking and animation for model-based-coded video,Proc. ICASSP2000, 2000.
[2] S. Birchfield, Derivation of Kanade-Lucas-Tomasi
Tracking Equation, unpublished, May 1996. Available on
the website: http://vision.stanford.edu/~birch
[3] M. J. Black and P.Anandan, The robust estimation of
multiple smooth motions: parametric and piecewise
smooth flow fields, Computer Vision and Image
Understanding, pp 75-104, January 1996.
[4] M.J. Black, D.J. Fleet, Y. Yacoob, A framework for
modeling appearance change in image sequences, Proc.
Intl. Conf. Computer Vision, 1998.
[5] K. Grauman, M. Betke, J. Gips and G. Bradski,
Communication via Eye Blinks - Detection and Duration
Analysis in Real Time, Proc CVPR 2001, 2001.
[6] B. D. Lucas and T. Kanade, An iterative image
registration technique with an application to stereo vision,
International Joint Conference on Artificial Intelligence,
1981.
[7] T. Nakano, K Sugiyama, M. Mizuno and S. Yamamoto,
Blink measurement by image processing and application to
warning of drivers drowsiness in automobiles, IEEE
Intelligent Vehicles, 1998.
8/9/2019 Seminar 8th Eye Blink
20/20
[8] P.Smith, M.Shah, N. daVitoria, Monitoring head-eye
motion for driver alertness with one camera, Pattern
Recognition, 2000.
[9] C. Tomasi and T. Kanade, Detection and Tracking of
Point Features, Carnegie Mellon University Technical
Report CMU-CS-91-132, April 1991.
[10] H. Ueno, M. Kaneda and M. Tsukino, Development
of Drowsiness Detection System, Proc. Conf. Vehicle
Navigation & Information Systems Conference, 1994.
[11] K.Yano, K.Ishihara, M. Makikawa, H. Kusuoka,
Detection of eye blinking from video camera with dynamic
ROI fixation, IEEE Trans. SMC, Vol 6, pp 335-339,
1999.
Recommended