1

Vision-based systems for driver

monitoring and de-identification

Human Sensing

Fernando De la Torre

A dream

Outline

• Driver distraction detection

• System

• Evaluation

• Video de-identification

• System

• Preliminary evaluation

• Conclusions

Outline

• Driver distraction detection

• System

• Evaluation

• Video de-identification

• System

• Preliminary evaluation

• Conclusions

Distracted driving has become a large problem due to the increasing use of cell phones

National Safety Council: 23% of all crashes involve cell phone use

Amount of drivers texting while driving:

• U.K: 45% at least one time

• Australia: 17 % regularly

• United States: 27% adults once at least

Related problems:

• Amount of time with Eyes-Off-The-Road (EOTR) increase by up to 400%

• 28% more lane excursions and 140% more incorrect lane changes

• Chances of a crash increase by up to 8 to 23 times

Recent study in U.K. concludes that

Texting while driving could be more dangerous than driving drunk

“Mobile Phone: A growing problem of driver distraction”, World Health Organization (2011)

Problem of texting while driving

5

Goals:

� Driver distraction detection, e.g.,

Eyes-Off-Road (EOR)

Challenges

� Low cost camera

� Pre-defined camera location

� Abrupt change in pose & illumination

� Night and day

� Real-time gaze and head pose

tracking

� Glasses support

6

Goals and Challenges

Goals:

� Driver distraction detection, e.g.,

Eyes-Off-Road (EOR)

Challenges

� Low cost camera

� Pre-defined camera location

� Abrupt change in pose & illumination

� Night and day

� Real-time gaze and head pose

tracking

� Glasses support

7

Webcam $60

Goals and Challenges

Goals:

� Driver distraction detection, e.g.,

Eyes-Off-Road (EOR)

Challenges

� Low cost camera

� Pre-defined camera location

� Abrupt change in pose & illumination

� Night and day

� Real-time gaze and head pose

tracking

� Glasses support

8

Goals and Challenges

Goals:

� Driver distraction detection, e.g.,

Eyes-Off-Road (EOR)

Challenges

� Low cost camera

� Pre-defined camera location

� Abrupt change in pose & illumination

� Night and day

� Real-time gaze and head pose

tracking

� Glasses support

9

Goals and Challenges

Goals:

� Driver distraction detection, e.g.,

Eyes-Off-Road (EOR)

Challenges

� Low cost camera

� Pre-defined camera location

� Abrupt change in pose & illumination

� Night and day

� Real-time gaze and head pose

tracking

� Glasses support

10

Goals and Challenges

Goals:

� Driver distraction detection, e.g.,

Eyes-Off-Road (EOR)

Challenges

� Low cost camera

� Pre-defined camera location

� Abrupt change in pose & illumination

� Night and day

� Real-time gaze and head pose

tracking

� Glasses support

11

Goals and Challenges

Goals:

� Driver distraction detection, e.g.,

Eyes-Off-Road (EOR)

Challenges

� Low cost camera

� Pre-defined camera location

� Abrupt change in pose & illumination

� Night and day

� Real-time gaze and head pose

tracking

� Glasses support

12

Goals and Challenges

13

EOR detection

from head pose Prediction

System Overview

driver imagefacial landmark

trackinghead pose gaze

Estimation

14

Eyes Off the Road (Ray Tracing)

Eyes Off the Road Estimation:

The EOR estimation is based on a 3D ray tracing method that uses the geometry of the scene.

Outline

• Driver distraction detection

• System

• Evaluation

• Video de-identification

• System

• Preliminary evaluation

• Conclusions

16

Results

VTTI has developed a protocol to evaluate the performance of the system:

• Extended Off-Road glances (2 sec)

• Brief check glances

Locations of interest selected:

Evaluation protocol

17

18

Results

Extended Off-Road glances (2 sec):

CMU System = System D

IntraFace

Download it!Google: itunes intraface

ghttps://itunes.apple.com/us/app/intraface/id937424937?mt=8

Outline

• Driver distraction detection

• System

• Evaluation

• Video de-identification

• System

• Preliminary evaluation

• Conclusions

Project Scope• Video De-identification in the Automobile Environment

• Remove person-specific facial features

• Preserve head pose, gaze and facial expression

Project Scope

• Video de-identification using Personalized FAT

Mask Face

Project Scope• Our approach: Personalized Facial Action Transfer (FAT)

Results on SHRP2 videos

Results on SHRP2 videos

Outline

• Driver distraction detection

• System

• Evaluation

• Video de-identification

• System

• Preliminary evaluation

• Conclusions

Evaluate identity protection

• Face recognition experiments on the images in CMU Multi-PIE

database (multipie.org) and frames in NDS videos.

• FAT produced the lowest recognition accuracies: best identity protection.

• An average face was used as mask, no distinctive facial features.

• Face recognition classifiers were trained and tested in the same image sets

or videos, an unlikely case in real applications.

Results:

Facial Action Unit Coding

Nose Wrinkler (AU9)Upper lid raiser (AU5) Lip Tightener (AU23)Outer Brow Raiser (AU2)

Evaluate data utility

• Head pose estimation experiments on the images in CMU Multi-

PIE database (multipie.org) and frames in NDS videos

• Facial Action Unit recognition on frames in NDS videos

• FAT preserved the most recognizable AU#12 in de-identified frames.

• Tracker needs to be improved for landmarks on the mouth.

• The head pose angles were preserved in the FAT de-identified

frames

Results:

In Progress• De-identify a NDS video using an average mask face

In Process• De-identify a NDS video using other methods

Conclusions

• It is a difficult problem

– Low spatial and temporal resolution

– Noisy camera (specially at night)

– Large changes in pose

• Preliminary results are encouraging, especially during the day

• Currently improving

– Tracking at night

– Stabilizing tracking in mouth and eye region

Thanks

Questions?

Human Sensing Laboratory

(http://humansensing.cs.cmu.edu)

Work supported by Federal Highway

Administration (FHWA)