Download pdf - Deep Learning for Self Driving Cars - Meetupfiles.meetup.com/19871959/...deep-learning-for-self-driving-cars.pdf · Deep Learning for Self Driving Cars. ... 5 seconds is the average

Lex Fridman: [email protected] Link: deeplearning.mit.edu

Deep Learning for Self Driving Cars


Course: Deep Learning for Self-Driving Cars

• Starts January 9, 2017

• All lecture materials will be made publicly available

• All in TensorFlow

• Two video game competitions open to the public:

• Rat Race: Winning the Morning Commute

• 1000 Mile Race in American Truck Simulator

http://deeplearning.mit.edu


Semi-Autonomous Vehicle ComponentsExternal

1. Radar

2. Visible-light camera

3. LIDAR

4. Infrared camera

5. Stereo vision

6. GPS/IMU

7. CAN

8. Audio

Internal

1. Visible-light camera

2. Infrared camera

3. Audio


Self-Driving Car Tasks

• Localization and Mapping:Where am I?

• Scene Understanding:Where is everyone else?

• Movement Planning:How do I get from A to B?

• Driver State:What’s the driver up to?


Stairway to Automation

Ford F150

Tesla Model S

Google Self-Driving Car

Data-driven approaches can help at every step not just at the top.


Robotics at MIT: 31 Groups


Shared Autonomy:Self-Driving Car with Human-in-the-Loop

Teslas instrumented: 17

Hours of data: 5,000+ hours

Distance traveled: 70,000+ miles


Shared Autonomy:Self-Driving Car with Human-in-the-Loop


Driving: The Numbers(in United States, in 2014)

• All drivers: 10,658 miles(29.2 miles per day)

• Rural drivers: 12,264 miles

• Urban drivers: 9,709 miles

• Fatal crashes: 29,989

• All fatalities: 32,675

• Car occupants: 12,507

• SUV occupants: 8,320

• Pedestrians: 4,884

• Motorcycle: 4,295

• Bicyclists: 720

• Large trucks: 587

Miles Fatalities


Cars We Drive


Human at the Center of Automation:The Way to Full Autonomy Includes the Human

Ford F150 Tesla Model S Google Self-Driving Car

FullyHuman

Controlled

FullyMachine

Controlled


Human at the Center of Automation:The Way to Full Autonomy Includes the Human

• Emergency• Automatic emergency breaking (AEB)

• Warnings• Lane departure warning (LDW)

• Forward collision warning (FCW)

• Blind spot detection

• Longitudinal• Adaptive cruise control (ACC)

• Lateral• Lane keep assist (LKA)

• Automatic steering

• Control and Planning• Automatic lane change

• Automatic parking

Tesla Autopilot


Distracted Humans

• Injuries and fatalities:3,179 people were killed and 431,000 were injured in motor vehicle crashes involving distracted drivers(in 2014)

• Texts:169.3 billion text messages were sent in the US every month.(as of December 2014)

• Eye off road:5 seconds is the average time your eyes are off the road while texting. When traveling at 55mph, that's enough time to cover the length of a football field blindfolded.

What is distracted driving?

• Texting

• Using a smartphone

• Eating and drinking

• Talking to passengers

• Grooming

• Reading, including maps

• Using a navigation system

• Watching a video

• Adjusting a radio


4 D’s of Being Human:

Drunk, Drugged, Distracted, Drowsy

• Drunk Driving: In 2014, 31 percent of traffic fatalities involved a drunk driver.

• Drugged Driving: 23% of night-time drivers tested positive for illegal, prescription or over-the-counter medications.

• Distracted Driving: In 2014, 3,179 people (10 percent of overall traffic fatalities) were killed in crashes involving distracted drivers.

• Drowsy Driving: In 2014, nearly three percent of all traffic fatalities involved a drowsy driver, and at least 846 people were killed in crashes involving a drowsy driver.


In Context: Traffic Fatalities

Total miles driven in U.S. in 2014:

3,000,000,000,000 (3 million million)

Fatalities: 32,675(1 in 90 million)

Tesla Autopilot miles driven since October 2015:

200,000,000 (200 million)

Fatalities: 1




3,000,000,000,000 (3 million million)

Fatalities: 32,675(1 in 90 million)


200,000,000 (200 million)

Fatalities: 1




3,000,000,000,000 (3 million million)

Fatalities: 32,675

(1 in 90 million)


130,000,000 (130 million)

Fatalities: 1

We (increasingly) understand this

We do not understand this (yet)

We need A LOT of real-world semi-autonomous driving data!

Computer Vision + Machine Learning + Big Data = Understanding


Vision-Based Shared Autonomy

Face Video

• Gaze

• Emotion

• Drowsiness

Dash Video

• Hands on wheel

• Activity

• Center stack interaction

Instrument Cluster

• Autopilot state

• Vehicle state

Forward Video

• SLAM

• Pedestrians, Vehicles

• Lanes

• Traffic signs and lights

• Weather conditions

Supporting Sensors

• Audio

• CAN

• GPS

• IMU

Raw Video Data

Deep NetsBehavior Analysis

Semi-Supervised Learning

SharedAutonomy


Camera and Lens Selection

Fisheye: Capture full range of head, body movement inside vehicle.

2.8-12mm Focal Length: “Zoom” on the face without obstructing the driver’s view.

Logitech C920:On-board H264 Compression

Case for C-Mount Lens:Flexibility in lens selection


Semi-Automated Annotation for Hard Vision Problems

Hard = high accuracy requirements + many classes + highly variable conditions

Real Example: Gaze ClassificationTrade-Off: Human Labor vs Accuracy

Acc

ura

cy

Human Labor

100%

90%

0 ba


Big Data is Easy… Big “Compute” Is Hard

• Initial funding: $95M• 4,000 CPUs• 800 GPUs• 10 gigabit link to MIT








Visual Odometry

• 6-DOF: freed of movement• Changes in position:

• Forward/backward: surge

• Left/right: sway

• Up/down: heave

• Orientation:• Pitch, Yaw, Roll

• Source: • Monocular: I moved 1 unit

• Stereo: I moved 1 meter

• Mono = Stereo for far away objects• PS: For tiny robots everything is “far away” relative to inter-

camera distance


SLAM: Simultaneous Localization and MappingWhat works: SIFT and optical flow

Source: ORB-SLAM2 on KITTI dataset


Visual Odometry in Parts

• (Stereo) Undistortion, Rectification

• (Stereo) Disparity Map Computation

• Feature Detection (e.g., SIFT, FAST)

• Feature Tracking (e.g., KLT: Kanade-Lucas-Tomasi)

• Trajectory Estimation• Use rigid parts of the scene (requires outlier/inlier detection)

• For mono, need more info* like camera orientation and height of off the ground

* Kitt, Bernd Manfred, et al. "Monocular visual odometry using a planar road model to solve scale ambiguity." (2011).


End-to-End Visual Odometry

Konda, Kishore, and Roland Memisevic. "Learning visual odometry with a convolutional network." International Conference on Computer Vision Theory and Applications. 2015.


Semi-Autonomous Tasks for a

Self-Driving Car Brain






Object Detection

• Past approaches: cascades classifiers (Haar-like features)

• Where deep learning can help:recognition, classification, detection

• TensorFlow: Convolutional Neural Networks


Computer Vision:

Object Recognition / Classification

Lex Fridman: [email protected] Link: deeplearning.mit.eduSource: Long et al. Fully Convolutional Networks for Semantic Segmentation. CVPR 2015.

Original Ground Truth FCN-8

Computer Vision:

Segmentation


Computer Vision:

Object Detection


Full Driving Scene Segmentation

Fully Convolutional Network implementation: https://github.com/tkuanlun350/Tensorflow-SegNet

https://github.com/tkuanlun350/Tensorflow-SegNet


Road Texture and Condition from Audio

Recurrent Neural Network (LSTM) implementation: https://www.tensorflow.org/versions/r0.11/tutorials/recurrent/index.html

https://www.tensorflow.org/versions/r0.11/tutorials/recurrent/index.html








• Previous approaches: optimization-based control

• Where deep learning can help: reinforcement learning

Deep Reinforcement Learning implementation:https://github.com/nivwusquorum/tensorflow-deepq

https://github.com/nivwusquorum/tensorflow-deepq


Deep Q-Learning

Implementation: https://github.com/harvitronix/reinforcement-learning-car

Google DeepMind approach with Atari:

Take only state as input and output: Q-value for each action)


Deep Q-Learning



• Localization:Where am I?

• Object detection:Where is everyone else?

• Movement planning:How do I get from A to B?

• Driver state:What’s the driver up to?


Drive State Detection:

A Multi-Resolutional View

39

GazeClassification

BlinkRate

BlinkDuration

HeadPose

EyePose

PupilDiameter

MicroSaccades

Increasing level of detection resolution and difficulty

BodyPose

BlinkDynamics

MicroGlances

CognitiveLoad

Drowsiness


Driver Gaze Classification

TensorFlow: Convolutional Neural Network https://github.com/mpatacchiola/deepgaze

https://github.com/mpatacchiola/deepgaze


Perfectly Synchronized Data


Gaze Region and Autopilot State


Driver Emotion

TensorFlow: CNN (for video) and RNN (for audio)https://github.com/synapse-uf/emotion-recognition

https://github.com/synapse-uf/emotion-recognition


Autonomous Driving: End-to-End

MagicHappens


Stairway to Automation

Ford F150

Tesla Model S

Google Self-Driving Car

Training Dataset

Testing Dataset









• 9 layers• 1 normalization layer

• 5 convolutional layers

• 3 fully connected layers

• 27 million connections

• 250 thousand parameters


End-to-End Driving Implementation

TensorFlow code with documentation will be made available at: deeplearning.mit.edu


End-to-End Driving with TensorFlow



Build the Model: Input and Output

def weight_variable(shape):

initial = tf.truncated_normal(shape, stddev=0.1)

return tf.Variable(initial)

def bias_variable(shape):

initial = tf.constant(0.1, shape=shape)

return tf.Variable(initial)

def conv2d(x, W, stride):

return tf.nn.conv2d(x, W, strides=[1, stride, stride, 1],

padding='VALID')

x = tf.placeholder(tf.float32, shape=[None, 66, 200, 3])

y_ = tf.placeholder(tf.float32, shape=[None, 1])

x_image = x


Build the Model: Convolutional Layers

#first convolutional layer

W_conv1 = weight_variable([5, 5, 3, 24])

b_conv1 = bias_variable([24])

h_conv1 = tf.nn.relu(conv2d(x_image, W_conv1, 2) + b_conv1)

#second convolutional layer



h_conv2 = tf.nn.relu(conv2d(h_conv1, W_conv2, 2) + b_conv2)

#third convolutional layer




#fourth convolutional layer




#fifth convolutional layer





Build the Model: Fully Connected Layers# fully connected layer 1

W_fc1 = weight_variable([1152, 1164])

b_fc1 = bias_variable([1164])

h_conv5_flat = tf.reshape(h_conv5, [-1, 1152])

h_fc1 = tf.nn.relu(tf.matmul(h_conv5_flat, W_fc1) + b_fc1)

keep_prob = tf.placeholder(tf.float32)

h_fc1_drop = tf.nn.dropout(h_fc1, keep_prob)

# fully connected layer 2



h_fc2 = tf.nn.relu(tf.matmul(h_fc1_drop, W_fc2) + b_fc2)












#Output



y = tf.mul(tf.atan(tf.matmul(h_fc4_drop, W_fc5) + b_fc5), 2)


Train the Model

sess = tf.InteractiveSession()

loss = tf.reduce_mean(tf.square(tf.sub(model.y_, model.y)))

train_step = tf.train.AdamOptimizer(1e-4).minimize(loss)

sess.run(tf.initialize_all_variables())

saver = tf.train.Saver()

for i in range(int(driving_data.num_images * 0.3)):

xs, ys = driving_data.LoadTrainBatch(100)

train_step.run(feed_dict={model.x: xs, model.y_: ys, model.keep_prob: 0.8})

if i % 10 == 0:

xs, ys = driving_data.LoadValBatch(100)

print("step %d, val loss %g"%(i, loss.eval(feed_dict={

model.x:xs, model.y_: ys, model.keep_prob: 1.0})))

if i % 100 == 0:

if not os.path.exists(LOGDIR):

os.makedirs(LOGDIR)

checkpoint_path = os.path.join(LOGDIR, "model.ckpt")

filename = saver.save(sess, checkpoint_path)

print("Model saved in file: %s" % filename)


Run the Model

import tensorflow as tf

import scipy.misc

import model

import cv2

sess = tf.InteractiveSession()

saver = tf.train.Saver()

saver.restore(sess, "save/model.ckpt")

img = cv2.imread('steering_wheel_image.jpg',0)

rows,cols = img.shape

smoothed_angle = 0

cap = cv2.VideoCapture(0)

while(cv2.waitKey(10) != ord('q')):

ret, frame = cap.read()

image = scipy.misc.imresize(frame, [66, 200]) / 255.0

degrees = model.y.eval(feed_dict={model.x: [image], model.keep_prob:

1.0})[0][0] \

* 180 / scipy.pi

cv2.imshow('frame', frame)

smoothed_angle += 0.2 * pow(abs((degrees - smoothed_angle)), 2.0 / 3.0) * \

(degrees - smoothed_angle) / abs(degrees - smoothed_angle)

M = cv2.getRotationMatrix2D((cols/2,rows/2),-smoothed_angle,1)

dst = cv2.warpAffine(img,M,(cols,rows))

cv2.imshow("steering wheel", dst)

cap.release()

cv2.destroyAllWindows()


End-to-End Driving with TensorFlow
