Beyond Pedestrian Detection: Deep Neural Networks Level-Up...

Ikuro Sato, Hideki Niihara

R&D Group, Denso IT Laboratory, Inc.

Beyond Pedestrian Detection:

Deep Neural Networks Level-Up Automotive Safety

Advanced Driver Assistance Systems (ADAS)

pedestrian detection with infrared camera forerunning car detection with radar

Images taken from DENSO’s TV ad.

What would a skilled driver do?

Today’s Technology

• Only to detect pedestrians with a camera

– limited autonomous maneuvers possible, e.g., slows down to stop

today’s industrial technology

The Goal

• Able to extract richer information about pedestrians

– various autonomous maneuvers possible like a skilled driver

future industrial technology

old-aged man finishing crossing

running toward the vehicle

man holding smartphone, crossing intersection

Three Technical Challenges

• multiclass classification

– Need to identify various annotative info on top of binary classification

• visual feature design

– Who knows what visual features best separate a man with a

smartphone from a man without?

• real-time processing

Promising Tools

• multiclass classification

– need to identify various “states” on top of binary classification

• visual feature design

– Who knows what visual features best separate a man with a cell phone

from a man without?

• real-time processing

Deep Neural Networks

GPGPUs

• Why Deep Neural Networks?

• Why GPGPUs?

• Demo Setup

Basics

• Neural Network: one of the machine learning algorithms for

classification and/or regression

input 𝑥

hidden ℎ

simple Neural Network

By tuning 𝑊𝑘, one obtains an arbitrary function.

weight 𝑊1

ℎ = 𝑔 𝑊1𝑥

𝑦 = 𝑔 𝑊2ℎ

𝑔: differentiable nonlinear function

(simplified)

output 𝑦

Basics

• Neural Network: one of the machine learning algorithms for

classification and/or regression

input 𝑥 output

hidden ℎ

simple Neural Network

By tuning 𝑊𝑘, one obtains an arbitrary function.

weight 𝑊1

Can have arbitrary many units

multiclass

• Convolutional Neural Network (CNN) comprising

– Convolutional layer(s) local feature extraction

– Pooling layer(s) dimensionality reduction

– Fully-connected layer(s) classification/regression

simple CNN

Neural Networks for Visual Tasks

input image C P F F

• Convolutional Neural Network (CNN) comprising

– Convolutional layer(s) local feature extraction

– Pooling layer(s) dimensionality reduction

– Fully-connected layer(s) classification/regression

Neural Networks for Visual Tasks

input image C P F F

visual features automatically extracted

simple CNN

• Transforming data vectors at multiple layers help to increase the

level of abstraction of visual contents (Bengio, 2009).

Why is “Deep” Architecture Advantageous?

C P P P F F F C C

edges set of segments parts classifier

Deep CNN

• Why GPGPUs?

• Demo Setup

Major Reason

• Classification can be boosted.

– Deep CNN can run at real-time with a compact Tegra K1 processor.

Deep CNN is very GPU friendly!

A Deep CNN typically has a small number of sequential operations, each of which is dominated by highly-parallelizable, massive sums-of-products.

Minor Reasons

– Deep CNN can run at real-time with compact Tegra K1 processor.

• Learning can be boosted.

– It can take a month without GPUs. Takes only several days with GPUs.

• Development can be boosted.

– Capability of processing generic programing language (i.e., CUDA)

makes R&D process efficient.

– No more fixed-points!

Minor Reasons

– Deep CNN can run at real-time with compact Tegra K1 processor.

• Learning can be boosted.

– It can take a month without GPUs. Takes only several days with GPUs.

• Development can be boosted.

– Capability of processing generic programing language (i.e., CUDA)

makes R&D process efficient.

– No more fixed-points!

• Why GPGPUs?

• Demo Setup

Problem Statement

• Real-time pedestrian detection with depth, height, and body

orientation estimations

– requires discrete-valued classification and real-valued regression tasks

Deep CNN Model

• A single deep CNN with 9 (processing) layers

– 3 Convolutional + (Max) Pooling layers

– 3 Fully-connected layers

16 maps 16 maps

16 maps

16 units

32 units 140x60

gray image

Deep CNN Model

• Output layer contains:

1 binary classification unit for pedestrian/background

1 regression unit for relative x-coord. of the body center

1 regression unit for relative y-coord. of the top of the head

1 regression unit for relative y-coord. of the toe

4 classification units for body orientations

𝑦head

𝑦toe

𝑥center

scanning window

Deep CNN Model

• Output layer contains:

1 binary classification unit for pedestrian/background

1 regression unit for relative x-coord. of the body center

1 regression unit for relative y-coord. of the top of the head

1 regression unit for relative y-coord. of the toe

4 classification units for body orientations

𝑦head

𝑦toe

scanning window converted to depth & height

𝑥center

Processor

• NVIDIA’s Jetson platform equipped with Tegra K1 processor

– 192 Kepler CUDA cores

– 200 GFLOPS (automotive K1)

– memory shared between GPU and CPU (ARM)

– low power consumption for automotive applications, e.g., ADAS

Jetson

Processor

• NVIDIA’s Jetson platform equipped with Tegra K1 processor

– 192 Kepler CUDA cores

– 200 GFLOPS (automotive K1)

– memory shared between GPU and CPU (ARM)

– low power consumption for automotive applications, e.g., ADAS

Jetson Deep CNN

Take-Home Messages

1. Extracting rich information about pedestrians will be

one of the key technologies for automotive safety.

2. Deep Neural Networks are the best algorithms for

visual recognition tasks.

3. GPGPUs enable real-time processing.

– Demonstrated that Tegra K1 indeed does!

Beyond Pedestrian Detection: Deep Neural Networks Level-Up...

Documents

Introduction to Deep Neural Networks - Deep Learning · Introduction to Deep Neural Networks 0. Logistics Spring 2020 1. Neural Networks are taking over! •Neural networks have become

Arti cial neural networks predicting pedestrian dynamics ...€¦ · Arti cial neural networks predicting pedestrian dynamics in complex buildings Antoine Tordeuxa, Mohcine Chraibi

DeepIoT: Compressing Deep Neural Network Structures … · DeepIoT: Compressing Deep Neural Network ... sensing-related tasks on Intel Edison ... Compressing Deep Neural Network Structures

Deep Neural Networks - uchile.cl

Deep Learning for Neural Networks - viXra · Deep Learning for Neural Networks Today, deep neural networks with different architectures, such as convolutional, recurrent and autoencoder

Scalable Object Detection using Deep Neural … Object Detection using Deep Neural Networks ... puter vision. A common paradigm to ... Scalable Object Detection using Deep Neural Networks

Deep Multi-State Dynamic Recurrent Neural Networks ...papers.nips.cc/paper/9594-deep-multi-state-dynamic-recurrent-neural... · Deep Multi-State Dynamic Recurrent Neural Networks

Representation Learning and Deep Neural Networks - …didawiki.di.unipi.it/.../computational-neuroscience/5-deep-hand.pdfRepresentation Learning and Deep Neural Networks Davide Bacciu

Multispectral Deep Neural Networks for Pedestrian Detectionszhang16/paper/BMVC16_multispectral.pdfdetector Faster R-CNN on KAIST multispectral pedestrian dataset (KAIST) [17]. Second,

Pedestrian detection using Doppler radar and LSTM neural

Deep Learning in Neural Networks: An Overviejuergen/DeepLearning28May2014.pdf · Deep Learning in Neural Networks: An Overview ... 1 Introduction to Deep Learning ... Neural Fitted

Practical Deep Neural Networks - DGYrt.dgyblog.com/res/dlworkshop/introduction.pdf · Practical Deep Neural Networks ... ' CS231n Convolutional Neural Networks for Visual Recognition

DEEP LEARNING AND NEURAL NETWORKS: BACKGROUND AND …wcohen/10-605/deep-1.pdf · DEEP LEARNING AND NEURAL NETWORKS: BACKGROUND AND HISTORY 1. ... Convolutional Neural Networks for

Debugging deep neural networks

Multispectral Deep Neural Networks for Pedestrian

ImageNet Classification with Deep Deep Convolutional …rita/visual_recog_course/talks/ImageNet... · ImageNet Classification with Deep Deep Convolutional Convolutional Neural Neural

Deep neural networks

Deep Neural Decision Forests

Artificial Neural Networks and Deep Learningchrome.ws.dei.polimi.it/images/a/ae/AN2DL_02_2021_Prceptron_2_Fee… · Artificial Neural Networks and Deep Learning . 2 «Deep Learning

Deep Neural Networks in Machine Translation: An …perpustakaan.unitomo.ac.id/repository/Deep Neural...Deep Neural Networks in Machine Translation: An Overview Jiajun Zhang and Chengqing