Machine Learning for Autonomous Driving
Senior Analysis Engineer at Volvo Cars/Zenuity & Adjunct Docent at Chalmers
Stockholm (KTH), 2017-04-03
• Develop software for autonomous driving and driver assistance systems
• Autoliv and Volvo Cars will own Zenuity50/50
• Starting with 200 employees from Autoliv and Volvo Cars
• Volvo Cars and Autoliv will license and transfer the intellectual property for their ADAS systems to Zenuity
• Headquartered in Gothenburg with additional operations in Munich, Germany, and Detroit, USA
CEO: Dennis Nobelius
AD and AI among hottest trends
Gartner's 2016 Hype Cycle for Emerging TechnologiesFörarlöst i
praktiken Ai i allt
Top 10 trends from Nyteknik
Autonomous Driving (AD)
ML in the era of AD
• Interest in ML in the IV/ITS community
• Some examples
Drive Me: Self driving cars for sustainable mobility
• Growing mega cities
• Air quality major health problem
• Traffic accident global health issue
• Time for commuting
• Desire for time efficiency
• Desire for constant connectivity
SHAPING THE FUTURE MOBILITY
AD will be important for a sustainable mobility• Improved traffic safety• Improved environmental outcomes• Regain time
Volvo vision 2020
No one should be killed or seriously injured
in a new Volvo car by 2020
It was always about freedom
And it’s still about freedom
joint effort and proper reseach
Global challenges – Demand a joint effort
Drive Me – Nordic model of collaboration
Research platform – How autonomous cars can contribute to a sustainable
Pilots with real customers in real traffic
The pilots – all about learning
• Traffic environments
• Customer preferences
• Exporting the Nordic model
Gothenburg – proof of concept
China and London – verify our technology
well-defined commute highways
• Customer focus
• Risk Management
F r u s t r a t i n g c o m m u t eN e i g h b o r h o o d C l o s e t o w o r k
Your neighborhood, children,
no lane markings, roundabouts, ...Traffic lights, pedestrians, bicyclists,
Well defined use case on city highways
Pilot Assist versus Autopilot
• Driver is responsible, should monitor and supervise
• Driver responsible to intervene whenever needed
• Limitations: Lane markings, road design, oncoming objects,
pedestrians, animals, restrictions in steering/braking/acceleration
force that can be applied
Autopilot / UnSupervisedPilot assist / Supervised
• Manufacturer responsible
• Tested on and expects extreme situations
• Takes precautions, takes decisions
• Driver free to do something else
AD (Level 4)
AD (Level 1&2)
Injury ReductionCrash Avoidance
Net-CrashCrash After crash
We assume liability
“Volvo will assume liability for its
autonomous technology, when used properly.”
• Map data
• Cloud connection
• Traffic Control Centre
“Machine learning for sensor signal processing is in the core!”
Interest in ML, related to AD and ITS
Records in Google Scholar ITSC 2016 (out of 430 papers)
Improving the Results over THE Years
• Object detection rate in
• Moderate: Min. bounding
box height: 25 Px, Max.
occlusion level: Partly
truncation: 30 %
Grouping the ML Methods
Where is it trained?
• In the cloud
When is it trained?
How is it trained?
• Un-supervised (semi-supervised)
Where is it executed?
• Off-board (e.g., in the cloud)
When is it executed?
• Offline (or batch processing)
• System design
Modular systems vs Holistic Design
- Modular systems and ML to design individual components
- Holistic and end to end learning for driving
Sensor data Vehicle control
(1) Well-defined task-specific modules (2) Could have parallel modules
• Semantic segmentation and free space and drivable area detection
• Object detection and tracking and information (speed, heading, type, ...)
• Road information and geometry of the routes
• Sensor fusion
• Scene Semantics such as traffic and signs, turn indicators, on-road markings etc
• Maps and updating them over time
• Positioning and localization
• Path planning
• Driving policy learning and decision making
• Other road user behavior analysis such as intention prediction
• Driver monitoring
Holistic and end to end learning
• First implemented around 28 years ago in the
• Number of parameters<50,000
• 9 layers (5 convolutional and 3 fully
• 250 000 parameters
• Dean A. Pomerleau, ”ALVINN, an autonomous land vehicle in a neural network”., No. AIP-77, CMU, 1989.
• Mariusz Bojarski, et al. "End to end learning for self-driving cars“, arXiv, Apr. 2016.
• Christopher Innocenti, Henrik Lindén, “Deep Learning for Autonomous Driving: A Holistic Approach for Decision Making”, thesis.
• Requirements: setting safety scope
for function and AD test scenarios
• Test methods• Test track
• Test in real traffic and expeditions
• Virtual testing and simulations
• Analyzing logged data• Need to have tools such as reference system
• Need to have advanced analysis methods
• Nidhi Kalra et al. "Driving to safety: How many miles of driving would it take to demonstrate autonomous vehicle reliability?." Transportation
Research Part A: Policy and Practice 94 (2016): 182-193.
(1) Accuracy (2) Speed
• Ross Girshick et al., “Rich feature hierarchies for accurate object detection and semantic segmentation”, CVPR, 2014.
• Shaoqing Ren et al., “Faster R-CNN: Towards real-time object detection with region proposal networks”, NIPS, 2015.
• Jifeng Dai et al., “R-FCN: Object Detection via Region-based Fully Convolutional Networks”, arXiv, Jun., 2016.
• Donal Scanlan, Lucia Diego, “Robust vehicle detection using convolutional networks“, Master’s thesis, ongoing.
• Jonathan Long et al., “Fully Convolutional Networks for Semantic Segmentation”, CVPR 2015.
• Jifeng Dai et al., “Instance-aware Semantic Segmentation via Multi-task Network Cascades”, arXiv, 2015
• Vijay Badrinarayanan et al., “SegNet: A Deep Convolutional Encoder-Decoder Architecture for Scene Segmentation”, PAMI 2017
• Christian Lipski et al. "A fast and robust approach to lane marking detection and lane tracking“, SSIAI, 2008.
• Florian Janda et al., "A road edge detection approach for marked and unmarked lanes based on video and radar“, FUSION, 2013.
• Jihun Kim et al., “Robust lane detection based on convolutional neural network and random sample consensus”, ICONIP, 2014.
• Bei He et al., “Accurate and Robust Lane Detection based on Dual-View Convolutional Neutral Network”, IV, 2016.
• Gabriel L. Oliveira et al., "Efficient deep models for monocular road segmentation“, IROS, 2016.
Traffic Sign Recognition
German Traffic Sign Recognition Benchmark (GTSRB) [Stallkamp 2012]
• Johannes Stallkamp et al. "Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition“, Neural networks, 2012.
• Pierre Sermanet et al., "Traffic sign recognition with multi-scale convolutional networks“, IJCNN, 2011.
• Konstantinos Mitritsakis, “Study Real World Traffic Environment Using Street View”, Master’s thesis, 2016.
Interacting with Humans
• Eshed Ohn-Bar et al., “Looking at Humans in the Age of Self-Driving and Highly Automated Vehicles”, Trans. on IV, 2015
• Kevan Yuen et al. ,”Looking at Faces in a Vehicle: A Deep CNN Based Approach and Evaluation”, ITSC, 2016.
• Akshay R. Siddharth et al., “Driver Hand Localization and Grasp Analysis: A Vision-based Real-time Approach”, ITSC, 2016.
• Tianchi Liu et al. "Driver distraction detection using semi-supervised machine learning." IEEE Transactions on ITS, 2016.
Road Friction Estimation from Connected Vehicles Data
• Prediction using both
historical friction data from the
connected cars and data from
• Busy roads
• Missing data
• Ghazaleh Panahandeh, Erik Ek, Nasser Mohammadiha, “Supervised Road Friction Prediction from Fleet of Car Data”, IV 2017.
Driver Route and Destination Prediction
• History of driving for
• Use of metadata such as
driver id, the number of
passengers, time-of-day, day-
• Destination clustering
• Ghazaleh Panahandeh, “Driver Route and Destination Prediction”, IV 2017
• Pruning or quantizing weights
• Devising new and smaller
• Forrest N. Iandola et al. "SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and< 0.5 MB model size“, arXiv, Nov. 2016
• Michael Treml et al., “Speeding up Semantic Segmentation for Autonomous Driving”, NIPS Workshop, 2016
• Adam Paszke et al. "ENet: A deep neural network architecture for real-time semantic segmentation “, arXiv, Jun. 2016
• Song Han et al., “Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding” arXiv , 2015.
• Alireza Aghasi et al., “Net-Trim: A Layer-wise Convex Pruning of Deep Neural Networks”, arXiv, Nov. 2016.
And many more ...
• Deep network understanding and interpretation
• Secure and privacy-preserving deep learning
• Transfer learning
• Pedestrian intention estimation
• Applications of generative adversarial networks
• Learning to attend
• Instance segmentation
• Object tracking
• Harsh weather conditions
• Traffic understanding such as brake light detection
• Benjamin Völz et al. “A data-driven approach for pedestrian intention estimation”, ITSC, 2016
• Arna Ghosh et al., “SAD-GAN: Synthetic Autonomous Driving using Generative Adversarial Networks”, arXiv, Nov. 2016
• Volodymyr Mnih et al. "Recurrent models of visual attention." NIPS, 2014.
Classification of 3D Point Clouds
• Patrik Nygren, Michael Jasinski, “A Comparative Study of Segmentation and Classification Methods for 3D Point Clouds”, Master’s thesis, 2016.
• Axel Bender, Elías Marel Þorsteinsson, “Object Classification using 3D Convolutional Neural Networks“, Master’s thesis, 2016.
• N. Mohammadiha, P Nygren, M. Jasinski, “A Comparison of Classification Methods for 3D Point Clouds”, Fast-zero 2017.
Ground truth for positioning
positioning in GPS-
• Scalability to new
• David Bennehag, Yanuar Nugraha, “Global Positioning inside Tunnels Using Camera Pose Estimation and Point Clouds”,
Master’s thesis, 2016.
Sensor comparison framework
• J. Florbäck, L. Tornberg, N. Mohammadiha, “Offline Object Matching and Evaluation Process for Verification of Autonomous
Driving”, ITSC, 2016.
• High expectations from ML to overcome some of the biggest
challenges in autonomous driving
• Successful applications of ML especially for perception already
being used in the industry
• New challenges arise in designing complete ML systems
(integration, updating, safety, interpretation...)
• Wide range of applications for ML from raw sensor data
processing to developing offline methods for verification purposes
• Industrial PhD position on “Reinforcement Learning for Autonomous Driving” at Zenuity/Chalmers
• Postdoc position on “Big Sensor Data Analysis for Verification of Autonomous Driving” at Zenuity/Chalmers
• Research Engineer position on “Big Sensor Data Analysis for Verification of Autonomous Driving” at Zenuity/Chalmers
Contact me: [email protected]
More positions on: