S7348: Deep Learning in Ford's Autonomous Vehicles

Bryan Goodman

Argo AI

9 May 2017

1

Today: examples from• Stereo image processing

• Object detection• Using RNN’s

• Motorsports

2

Ford’s 12 Year History in Autonomous Driving

Stereo Matching Problem

• Determining the correspondences in stereo images

• Calculating the disparities

• But what is the correct correspondence?

• Basic stereo matching algorithm− Compare pixels on the same

epipolar line in two images

− Choose the best match

3

Deep neural networks for stereo matching

• The brain can estimate the distance of an object using the visual information from two eyes.

• We can use deep neural networks

4

Right Stereo Camera

Deep Convolutional Neural Networks

Post-Processing

Left Stereo Camera

Distance Map Estimation

Proposed deep convolutional neural network• AV driving requires an intelligent distance map estimation, which filters out the

objects not of interest.• Network I

− General network

− Encoding and decoding layers

− Retain objects of interest in the training data sets

5

Co

nv1 Conv5

Co

nv2

Co

nv3

Co

nv6

Deco

nv6

Co

nv7

De

con

v7

Deco

nv8

Co

nv8

Deco

nv9

Encoder Decoder

Co

nv9

Loss Function

Deco

nv1

0

Co

nv1

0

Co

nv4

Proposed deep convolutional neural network II

− Specialized network

− Encoding and decoding layers

− The cross correlation layers force the network to look for correspondence on the epipolar line

− The weights in the encoding layers are shared

6

Co

nv1

L

Conv4L

Loss

Fun

ction

Encoder Decoder

Co

nv2

L

Co

nv3

L

Co

nv1

R

Conv4R

Co

nv2

R

Co

nv3

R

CC

5

Co

nv5

Co

nv6

CC

6D

econ

v6

Co

nv7

CC

7

Deco

nv7

Deco

nv8

Co

nv8

Deco

nv9

Co

nv9

Proposed deep convolutional neural network

• Cross correlation (CC) layer− Computes CC values between each pairs of

patches

− Outputs the CC values for each pair of patches

− Does not lose any information

• Loss function− In AV driving, closer objects are more important

than distant ones

− Assigns more weight to the closer objects

− The closer object distance is estimated more accurately

7

0.2 0.4 0.6 0.8 1

1

0.4

0.2

α

d

0.6

0.8

Performance on synthetic and real stereo data

• Synthetic data generation− Generate 14,000 pairs of RGB stereo images

− Synthetic distance maps are only generated for the objects of interest, e.g. cars or pedestrians

− Gaussian noise added to the stereo images

8

Performance on synthetic and real stereo data• Fine tuning with LIDAR data sets

− Project LIDAR point clouds onto the camera images

− The baseline and optic axes are not the same as the synthetic data

9

Left camera Right camera Network I Network II

1/2x

10

Comparing Manual Annotation to DNN Model

11

12

Detection Result Original Image Enhanced Contrast

Network’s detection outperforms human labelerin low-contrast areas

Pedestrian detection Pedestrian misdetection Detected, but not labeled

Introducing Recurrence in Detection and Tracking

• Use RNN’s to detect occluded objects• Remember location of static objects

• Predict location of non-static objects

13

Image 0

FeatureMap

RNN Conv

Image 1

FeatureMap

Image 2

FeatureMap

RNN Conv RNN Conv

Detector Detector Detector

Orange = ground truth; Green = model prediction

14

15

Classifying NASCAR images

The Ford team reviews pictures during the race

16

Classifying NASCAR images

Looking for damage and other performance indicators

Gap

17

Results –Boxing the Cars

Using ~2k images labeled

with boxes around the

vehicles, the model does

well detecting cars

18

Results –Boxing the Cars

Classifying NASCAR images

Next –determine car

number:labeled ~30k

images

Classifying NASCAR images

Outliers easy to find in review

Classifying NASCAR images

Human: ???Model: 78

Confidence: 0.999

Classifying NASCAR images

Human: ???Model: 42

Confidence: 0.985

Inspecting the Neural Network

23

Activated Filter Input Image

The Model is not a black box. We can see that it is detecting the numbers – important for robustness when the paint changes

Argo AI

• Argo AI is an artificial intelligence company, established to tackle one of the most challenging applications in computer science, robotics and artificial intelligence: self-driving vehicles

• Engineering hubs in Pittsburgh, Southeastern Michigan and the Bay Area of California

• For more information regarding Argo AI and its work, please talk to me at GTC or visit: www.argo.ai

24