Jack Weast

Principal Engineer, Chief Systems Engineer

Automated Driving Group, Intel

2

From the Intel Newsroom…

3

Levels of Automated Driving

Courtesy SAE InternationalRef: J3061

Data Center

4

Captured Sensor Data Analytics

In-Vehicle

Sensor Processing

Automated Driving Functions

Vehicle Simulation & Validation

Simplified End-to-End Automated Vehicle Architecture

Sensor Capture

Data

Vehicle Endpoint Management

What Makes an Automated Vehicle Work?

SensingSensor Fusion

Environment Modeling

HD-Map Correlation

Sensor Processing

ManeuveringPath

PlanningTrajectory

EnumerationLocalization

Decision Making

What is Happening in the Data Center?

Development of Automated Vehicles *starts* in the Data Center

There you store Data and Lots of it!!!

The first step towards developing your algorithms for autonomy

Example Data: Data

• Raw Sensor Data• Vehicle Bus Traffic• Environmental Conditions• Scenario Under Capture• Driver Name / Date / Time• More…

How Does the Data get There?

Test Fleet Vehicles produce an incredible amount of data

PetaBytes per hour!

Physical Ingestion is really the only option

Commercially Deployed Vehicles produce all the same amount of data

But it is used inside the vehicle

The trick is how to know when to capture “interesting” data for later cloud processing

Sneaker Net!

Intelligent AnomalyDetection

What else is Happening in the Data Center?

All of that uploaded data is used to trainour Deep Neural Networks

DNN’s support key use cases like

Pedestrian Detection

Traffic Sign Detection

Visual Object Detection

And, emerging use cases like

Driving Policy

Captured Sensor Data Analytics

Model TrainingMulti-node / IA-optimized Frameworks

Neural Network Design for Target Hardware, and usage

(Vision, Data Driven, etc.)

Big Data Analytics

Statistical Trends

Intel® Architecture (IA)

Simulation>than Real Time

Model Simulation & Verification

What is else is Happening in the Data Center?

All of that uploaded data is also used for ‘big data’ style statistical analysis

Statistical Analysis can provide:

Longitudinal usage patterns

Insight into vehicle performance

Personalization

The real value here is the intersection of Big Data and Deep Learning

Captured Sensor Data Analytics

Model TrainingMulti-node / IA-optimized Frameworks

Neural Network Design for Target Hardware, and usage

(Vision, Data Driven, etc.)

Big Data Analytics

Statistical Trends

Intel® Architecture (IA)

Simulation>than Real Time

Model Simulation & Verification

What else is Happening in the Data Center?

But how do I know if the algorithm works?

You need to test it on new, labeled data that was not previously used for training

Simulation >> faster than Real Time mechanism to test the proposed vehicle implementation

Crucial to do before deploying to the real world!

Simulation>than Real Time

Model Simulation & Verification

Intel® Architecture (IA)

Captured Sensor Data Analytics

Model TrainingMulti-node / IA-optimized Frameworks

Neural Network Design for Target Hardware, and usage

(Vision, Data Driven, etc.)

Big Data Analytics

Statistical Trends

Introducing Intel® Deep Learning SDKIntel DL Deployment Tool

• IMPORT Trained Model (trained on Intel or 3rd Party HW)

• COMPRESS Model for Scoring on Target Intel HW

• GENERATE Scoring HW-Specific Code (OpenCL*, C/C++)

• INTEGRATE with System SW / Application Stack & TUNE

• EVALUATE Results and ITERATE

configure_nn(fpga/cve,…)

allocate_buffer(…)

fpga_conv(input,output);

fpga_conv(…);

mkl_SoftMax(…);

mkl_SoftMax(…);

…

Optimized libraries & run-times (MKL-DNN, OpenVX, OpenCL)Data acquisition (sensors) and acceleration HW

Target Scoring Hardware Platform (physical or simulated)

MKL-DNN Optimized Machine Learning Frameworks

Intel DL Training Tool

• INSTALL / SELECT IA-Optimized Frameworks

• PREPARE / CREATE Dataset with Ground-truth

• DESIGN / TRAIN Model(s) with IA-Opt. Hyper-Parameters

• MONITOR Training Progress across Candidate Models

• EVALUATE Results and ITERATE

Intel® Xeon® Workstation or Cluster (local or cloud)

Intel® Architecture (IA)https://software.intel.com/en-us/deep-learning-sdk

Endpoint Management & Deployment

Connected Vehicles represent a significant Endpoint Management Challenge!

Our Goal is to deploy early and often, over the air, updated models and other driving policy preferences

Model Compression is the process of reducing the size (layers) of the network while retaining accuracy

Other methods could include some amount of re-training within the vehicle…

Vehicle EndpointManagement

Compression

Putting it All together: An End-to-End Architecture

Data Storage

Dataset Management

and Traceability

Data Center

13

Model TrainingMulti-nodeoptimized

Frameworks

Neural Network / Algorithm Design

In-Vehicle

Real Time Environment ModelingLocalization

Sensor Processing and FusionObject ID & Classification

Automated Driving FunctionsTrajectory Enumeration, Path

Planning, Selection & ManeuveringDriving Policy, Path Selection

CompressedDL Model

Captured Sensor Data

Real Time HD Map Updates

Data Formatting, Annotation

Annotation Tools

OTA SW /FW Updates

5G

Simulation>than Real Time

Model Simulation & Verification

An

om

aly

De

tectio

n

Endpoint ManagementGeographical Tracking, OTA Updates

Sneaker Net – Test Fleets

Big Data and Statistical Analytics

Introducing Intel® GO™ Automated Driving Solution

15

But what does all this mean for the In-Vehicle System Architecture?

Key Challenges in Enabling Autonomous Vehicles

Computational Performance and Power Efficiency

Data Center Compute in a limited power envelope

Real-time processing for vehicle control / collision avoidance

Safety and Securityfor increased levels of automation

CHALLENGES

Flexible & ScalablePlatform that scales across vehicle models from standard to luxury (price & performance)

17

The ROI Conundrum for Developing Custom Logic

1 2 3 4 5 6 7 8 9 10

$175$262

$439

$782

$1,605

* Development Cost Source: IBS

$ Millions

65nm 40nm 28nm 20nm 14nm

Revenue ROI Requiredto maintain R&D costs at 20% of Revenue

$35 $52 $88$156

$321

IC Development Costs*:IC Hardware

Software

Validation

18

FPGAs are Multi-function Accelerator Platforms

• Hardware accelerators provide breakthrough compute performance

• Re-configurable to adapt to a wide variety of workloads

• Performance-per-Watt gains with highly paralleled architecture

CPU

FPGA

Pre-distortion

Machine Learning

Sensor Fusion

Pedestrian Detection

Ra

dio

Au

tom

oti

ve

Da

ta C

en

ter

Accelerator

Accelerator

FPGA: Automated Driving Flexible Accelerator

Performance / Watt

Real-Time Processing

Flexible Accelerator

FPGA Value

FPGAAttributes

FPGAWorkload

Deep Learning

Grid Fusion

Path Planning

Sensor Fusion

Object Classification

DeterministicArchitecture

Hard FloatingPoint

EmbeddedMemory & DSP

ADHeterogeneous

Architecture

The MissingPuzzle Piece

Safety and SecuritySecurityArchitecture ∞

The Only Timeline That Matters Is When People Are Ready for Automated Driving

Trust means we feel:

• Safe

• Comfortable

• Confident

• “in control”

Without Trust, Automated Vehicle Adoption Won’t Happen

• Good listening

• Open dialogue

• Prompt action

How Do AVs Gain Our Trust?

Dr. Nicholas Epley

“The Mind in the

Machine” To the Car: A Name, Gender, and Voice.

So what are the key system elements of trust?

Visual, auditory, and motion sensors help the system “listen actively”—both inside and outside the vehicle.

SENSING

Touchscreens, voice controls, communication screens, and lighting give the vehicle multiple ways to communicate in multiple directions with passengers.

COMMUNICATING

Automated Vehicles must react quickly and effectively in response to driving conditions and routing requests.

RESPONDING TO CHANGES

• Key interactions and capabilities

• Technical implications

• Understanding what to optimize

We Are Building a

Trustworthy Platform.

Summary

Automated Driving is an End to End Architecture and Intel GO™ Automated Driving Solutions has you covered

Key Technical Challenges in Automated Vehicles can be solved through use of FPGAs as Flexible HW Accelerators for evolving workloads

Human afforded Trust is what will decide when we as humans are ready for automated vehicles

Incredibly scalable in-vehicle computing

• Wide range of reliable, available, and secure compute

• Collaboration with Wind River and others to deliver functionally safe OS, software, tools

• Layered security from chip to cloud with features rooted in hardware and support for secure over-the-air updates

Ideal combination of sequential and parallel computing

• Powerful and efficient Intel® Atom™ and Intel® Xeon® processors for sequential computing

• Arria® FPGAs for a powerful, cost-effective, scalable design platform

• Hardware acceleration technology for computer vision and machine/deep learning

• Suite of tools for automated driving software application developers

• Support for in-vehicle code and cloud development

• Machine learning/deep learning, computer vision algorithm development on Intel® architecture and accelerators

• Acceleration libraries, compilers, debuggers and IDE

Deep Learning, Computer Vision, and Sensor Fusion ToolsSensor Data Tool [NEW]

DL Training and Deployment ToolsIntel-optimized DL Frameworks

OpenVX* Kernel Library and Graph Builder

In-Vehicle Platform ToolsYocto* Recipe

Board Flashing Utility

FPGA & Heterogeneous Programming ToolsFPGA OpenCL

Full Stack Optimization Tools and LibrariesCompiler (ICC), JTAG debugger

Intel® Vtune™ Amplifier, Thread Check, Thread ProfilerPerformance and Threading Libraries (Intel® MKL, IPP, TBB)

• Industry’s first 5G-ready automotive platform

• Rapid development and testing of 5G solutions

• Target use cases:

• HD map downloads in real time

• HD content for in-vehicle infotainment

• Over-the-air updates

• Sensor uploads from vehicle, for machine learning

• Safety, smart intersections, cooperative driving

Introducing Intel® GO™ Automated Driving Solutions

Unmatched scalability and performance per power

Available 1H’17

1st 5G ready platform for automotive development

Available Feb 2017

Automotive Software Development Kit (SDK)

Intel Data Center Solutions

Automotive 5GPlatform

In-Vehicle Development

Platform for Automated Driving

>97% of servers deployed form machine learning workloads powered by Intel