32
Smart Cars for Safe Driving Prof. Dr. Dariu M. Gavrila Environment Perception Group Research and Advanced Engineering XXXII Jornadas de Automática, Sevilla, 9-9-2011

Smart Cars for Safe Driving - intranet.ceautomatica.es

  • Upload
    others

  • View
    2

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Smart Cars for Safe Driving - intranet.ceautomatica.es

Smart Cars for Safe Driving

Prof. Dr. Dariu M. GavrilaEnvironment Perception

Group Research and Advanced Engineering

XXXII Jornadas de Automática, Sevilla, 9-9-2011

Page 2: Smart Cars for Safe Driving - intranet.ceautomatica.es

2

We originally thought Machine Intelligence would look like

1956 "Forbidden Planet"

Robby the Robot (Flickr)

Page 3: Smart Cars for Safe Driving - intranet.ceautomatica.es

3

Then more recently, some suggested it would be more like

2004 iRobot 1983-2009 Terminator

Page 4: Smart Cars for Safe Driving - intranet.ceautomatica.es

4

when in fact, Machine Intelligence is already with us, and has a familiar embodiement

Page 5: Smart Cars for Safe Driving - intranet.ceautomatica.es

5

Driver Assistance in the current Mercedes Benz E-Class

Speed Limit

Adaptive

High Beam

PRE-SAFE

Lane Keeping

Blind Spot

Nightview

Plus

Attention

Page 6: Smart Cars for Safe Driving - intranet.ceautomatica.es

6

•Technology is rapidly expanding the capabilities of modern vehicles.

•One breakthrough development over the past few years is the emergence of

driver assistance systems.

•Use of sensor systems which continuously monitor vehicle surroundings and

interior, provide information to the driver, and even perform vehicle control.

•Help drivers operate their vehicles in a safe, comfortable, and energy-

efficient manner.

• Enables market differentiation for vehicle manufacturers

Driver Assistance

Page 7: Smart Cars for Safe Driving - intranet.ceautomatica.es

7

What got us here: Sensors

Better and cheaper.

Radars Cameras Laser Scanners

Page 8: Smart Cars for Safe Driving - intranet.ceautomatica.es

8

What got us here: Computational Power

CPU performance over time

MFlo

ps

in m

yve

hciles

10

Processing Power over Time

time

GFLOPS/MIPS

1990 2000 2010

100

200

300

400

500

ASIC

FPGA (Xilinx)

GPU (NVidia)

CPU (Intel)

Transputer/x86 P4

Core2Duo

G92

G80

G70

NV40

Virtex 5

Spartan3

Virtex 4

Tyzx

IQ2

Standford engine

3DIP

Processing Power over Time

time

GFLOPS/MIPS

1990 2000 2010

100

200

300

400

500

ASIC

FPGA (Xilinx)

GPU (NVidia)

CPU (Intel)

Transputer/x86 P4

Core2Duo

G92

G80

G70

NV40

Virtex 5

Spartan3

Virtex 4

Tyzx

IQ2

Standford engine

3DIP

*1,78/a

106

Prognosis 2030:

optimistic (1.78/a): 100 PFlops

pessimistic (1.41/a): 1 PFlops

(Still) exponentially increasing.

Page 9: Smart Cars for Safe Driving - intranet.ceautomatica.es

9

Next Challenge: Active Pedestrian Safety

Pedestrian are the most vulnerable traffic

participants. Children are particularly at risk.

Driver inattention and/or bad visibility are

important accident causes.

Worldwide fatalities of

pedestrians, bicyclists,

and motorcyclists (2006)

Source: Bosch Accident Research

Page 10: Smart Cars for Safe Driving - intranet.ceautomatica.es

10

Why is it difficult?

Large variation in pedestrian appearance (viewpoint, pose, clothes).

Dynamic and cluttered backgrounds.

Pedestrians can exhibit highly irregular motion.

Real-time processing required.

Stringent performance requirements (especially for emergency maneuvres).

Page 11: Smart Cars for Safe Driving - intranet.ceautomatica.es

11

Pedestrian System Architecture

Object

Classification

Object

Classification TrackingTracking

Driver Warning /

Vehicle Control

Driver Warning /

Vehicle Control

Path Prediction

&

Risk Assessment

Path Prediction

&

Risk Assessment

Obstacle Detection(Stereo, Flow, Radar)

Obstacle Detection(Stereo, Flow, Radar)

The benefit of object classification:

•improved detection reliability vs. obstacle detection only

•better path prediction: taking advantage of prior knowledge of

object class motion and additional object class-specific cues

•allows object class-specific driver warning and vehicle control strategies

D. M. Gavrila and S. Munder. Multi-Cue Pedestrian Detection and Tracking from a Moving Vehicle. IJCV 73(1), 2007.

S. Munder, C. Schnörr and D.M. Gavrila. Pedestrian Detection and Tracking Using a Mixture of View-Based Shape-Texture Models.

IEEE Trans. on Intelligent Transportation Systems, vol.9, nr.2, pp.333-343, 2008.

C. Keller, T. Dang, A. Joos, C. Rabe, H. Fritz, and D.M. Gavrila. Active Pedestrian Safety by Automatic Braking and Evasive Steering,

IEEE Trans. on Intelligent Transportation Systems, 2011

Page 12: Smart Cars for Safe Driving - intranet.ceautomatica.es

12

A. Wedel, C.Rabe, T. Vaudrey, T. Brox, U.Franke, D.Cremers.

“Efficient Dense Scene Flow from Sparse or Dense Stereo Data”. ECCV 2008.

stereo

stereo

optical flowoptical flow

ltyxI )1,,( −

ltvyuxI ),,( ++

ltvyuxI ),,( ++=

rtvyddduxI ),,( ++++=

rtvyddduxI ),,( ++++=

rtydxI )1,,( −+

3D Position and Motion for Every Pixel (Scene Flow)

Joint Optimization

Motion

time

t

time

t-1

Page 13: Smart Cars for Safe Driving - intranet.ceautomatica.es

13

Scene Flow

Page 14: Smart Cars for Safe Driving - intranet.ceautomatica.es

14

Pedestrian Classification – Experimental Studies

What features? E.g. Chamfer, Haar wavelets, HOG, and Local Receptive Field

What pattern classifier? E.g. SVM, Neural Networks

How to combine pattern classifiers? E.g. Cascading, Parallel (Sum/Max/Mixture)

How to deal with occlusion?

Haar wavelets + AdaBoost cascade

[Viola & Jones, 2005]

HOG features + linear SVM

[Dalal & Triggs, 2005]

Local receptive fields + NN

[Wöhler & Anlauf, 1999]

Page 15: Smart Cars for Safe Driving - intranet.ceautomatica.es

15

Daimler Pedestrian Benchmark Data Sets

1. Mono Pedestrian ClassificationS. Munder and D. M. Gavrila. An Experimental Study on Pedestrian Classification.

IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.28, no.11, pp. 1863-1868, 2006.

2. Multi-Modal / Occluded Pedestrian ClassificationM. Enzweiler, A. Eigenstetter, B. Schiele and D. M. Gavrila. Multi-Cue Pedestrian Classification with

Partial Occlusion Handling. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2010.

3. Mono/Stereo Pedestrian DetectionM. Enzweiler and D. M. Gavrila. Monocular Pedestrian Detection: Survey and Experiments.

IEEE Trans. on Pattern Analysis and Machine Intelligence, vol.31, no.12, pp.2179-2195, 2009.

C. Keller, M. Enzweiler, and D. M. Gavrila. A New Benchmark for Stereo-based Pedestrian Detection.

Proc. of the IEEE Intelligent Vehicles Symposium, Baden-Baden, Germany, 2011.

Training: 14400 peds. / 15000 non-peds.

Test: 9600 peds. / 10000 non-peds. All 18x36 pixel.

Available for download (Google)

Training: 15660 peds. / 6744 non-ped images

Test: 21790 images with 259 ped. trajectories

>130.000 samples (intensity, dense stereo,

dense flow), 48x96 pixel

1. 2. 3.

Page 16: Smart Cars for Safe Driving - intranet.ceautomatica.es

16

An intriguing question …

ROC performance improves with

enlarged training set. No saturation

effects (even) for N = 12.800

In fact, doubling training size

matters more than selecting

the best feature-classifier

combination.

How many image examples are needed to learn pedestrian appearance?

Manually labeling humans in images is time-consuming and tedious!Can we do better?

Page 17: Smart Cars for Safe Driving - intranet.ceautomatica.es

17

Generating Virtual Pedestrians

Shape

variation

Texture

variation

M. Enzweiler and D. M. Gavrila. A Mixed Generative-Discriminative Framework for Pedestrian Classification. CVPR 2008.

Page 18: Smart Cars for Safe Driving - intranet.ceautomatica.es

18

Mixed Generative-Discriminative Classification Framework

Enlarged training set significantly improved classification

performance (30% less false positives at equal true positive rate)

Meanwhile, current pedestrian classifier on-board vehicle uses

more than 1.5 million samples (“real” and “virtual”)

M. Enzweiler and D. M. Gavrila. A Mixed Generative-Discriminative Framework for Pedestrian Classification. CVPR 2008.

Page 19: Smart Cars for Safe Driving - intranet.ceautomatica.es

19

Pedestrian Detection - Daytime (Videoclip)

Page 20: Smart Cars for Safe Driving - intranet.ceautomatica.es

20

Pedestrian Detection – Nighttime (Videoclip)

Page 21: Smart Cars for Safe Driving - intranet.ceautomatica.es

21

Now with dense stereo …

Page 22: Smart Cars for Safe Driving - intranet.ceautomatica.es

22

EU WATCH-OVER (2008)85%

50 km/h

10

Pedestrian Recognition Performance (Historical Perspective)

Correctly recognizedpedestrians

Number of falsely recognized pedestrian trajectories per hour

100%

50%

10 10000 100100

EU SAVE-U (2005)

65%40 km/h

EU PROTECTOR (2003)

40%

600

30 km/h

N.B. # False alarms per hour << # Falsely recognized trajectories per hour

We need to get

somewhere here Source: EU Final Review WATCH-OVER

Page 23: Smart Cars for Safe Driving - intranet.ceautomatica.es

23

lateral position

(m)

longitudinal position

(m)

featu

re

dim

ensi

on

lateral position

longitudin

al posi

tion

(m)

Pedestrian Path Prediction by Trajectory Matching

Our approach

Use higher order model; match learned

trajectory “snippets” (segment of fixed length).

QRLCS (Hermes et al. IV’09) metric computes

similarity after alignment (translation/rotation).

Use of additional motion features.

Path prediction by extrapolation of matched

trajectory snippets (non-param. regression).

Use of particle filter representation.

C. Keller, C. Hermes and D. M. Gavrila. Will the pedestrian cross? DAGM 2011 Prize.

main mode

traj.

prediction

aligned snippet distribution

system trajectory

State-of-the-art path prediction: Kalman filter-

ing based on position detected bounding box.

Problem: first-order model does not capture

non-linearities well during sudden motion

changes.

Page 24: Smart Cars for Safe Driving - intranet.ceautomatica.es

24

Path Prediction

C. Keller, C. Hermes and D. M. Gavrila. Will the pedestrian cross? DAGM 2011 Prize.

Page 25: Smart Cars for Safe Driving - intranet.ceautomatica.es

25

Action Classification (Crossing or not)

Predicting the correct pedestrian’s

action with accuracy 80% is

reached:

•570 ms before a possible standstill

by the human (cyan).

•180 ms before a possible standstill

by the proposed system (black).

•only after the possible standstill by

the IMM-KF (pink).

Motion features help.

1 Frame ≈ 45ms

C. Keller, C. Hermes and D. M. Gavrila. Will the pedestrian cross? DAGM 2011 Prize.

Page 26: Smart Cars for Safe Driving - intranet.ceautomatica.es

26

World Premiere (2009): Automatic Evasion on Pedestrians

Page 27: Smart Cars for Safe Driving - intranet.ceautomatica.es

27

Automated Test Driving (Videoclip)

Source: Daimler Testing Department

Page 28: Smart Cars for Safe Driving - intranet.ceautomatica.es

28

Understand What Is What

Localize and classify objects in the environment

Background

Street

Pedestrian

Sky

Vision

Moving vehicle

Source: U. Franke

Page 29: Smart Cars for Safe Driving - intranet.ceautomatica.es

29

Driver Monitoring

Head / Face / Gaze Tracking Mindlab

Head / Face tracking using stereo vision

and Active Appearance Models

Driver intention estimation based on

head motion, gaze, and vehicle trajectory

Online EEG analysis of driver mental

state (work load, fatigue)

Use to objectively evaluate driver

assistance systems (Attention, IHC)

Page 30: Smart Cars for Safe Driving - intranet.ceautomatica.es

30

Automation Systems: Gradually Getting There

Not certifiable todayTraditional

driving

All on

Today‘s

ACC

Assisted 1st

Feet off

Short

takeover times

Assisted 2nd

Hands off

Moderate

takeover times

Autonomous 1st

Eyes off

Ability to

drive empty

Autonomous 2nd

Body out

Source: R. G. Herrtwich

Page 31: Smart Cars for Safe Driving - intranet.ceautomatica.es

31

Final Remarks

Driver assistance is experiencing a breakthrough: a first major deployment of

machine intelligence technology (sensing, reasoning, acting in physical environment).

Computer vision and machine learning play a central role.

Trend towards increased actuation of safety systemsDriver information � driver warning � “soft” vehicle actuation / driver-initiated “hard”

vehicle actuation � automatic “hard” vehicle actuation

Environment Perception is still the bottleneck. Need to

• recognize a wider set of traffic objects classes with better classification performance

• localize objects more accurately in 3D (perform segmentation and classification jointly).

• handle adverse visibility conditions

Future systems will fuse data from lots of sensors and build a precise 3D-

representation of the 360° car surrounding.

The progress in environment perception, driver monitoring, communication as

well as in precise 3D map data will bring us close to our vision of

accident free driving.

Page 32: Smart Cars for Safe Driving - intranet.ceautomatica.es

32

The best is yet to come!

Questions ?