S9554 - Fast Training of

Deep Neural Networks Using

Brain-Generated Labels

Sergey Vaisman, VP R&D, InnerEye

S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 2

“Without humans, artificial intelligence is stillpretty stupid..”

Image: The New York Times

(The Wall Street Journal)

S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 3

Data

Collection

Dataset

Annotation

Neural

Network

Design

Deployment

New Data

Neural

Network

Training

Optimization

Dataset

Exploration

AI Training Challenges

S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 4

0.4

0.5

0.6

0.7

0.8

0.9

1

0 20 40 60 80 100 120 140 160 180 200 220 240 260

AU

C(A

rea

Un

der

th

e C

urv

e)

Time [min]

Classification Performance

Brain In The Loop - Iterative AI Training Framework

S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 5

Image Classification Average Performance

10x times faster

Trained on NVIDIA GeForce GTX1080 ti GPU

S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 6

InnerEye – The CompanyCombining Human Intelligence with Artificial Intelligence

▪ Founded in 2014 - Technology spin-off from Israel’s The Hebrew and Ben-Gurion Universities

▪ Offices in Herzliya, Israel and Tokyo, Japan

▪ Over $6M of funding provided so far

▪ Products: Visual content review, AI Training and Validation, Connected Human

▪ Management Team:

Uri AntmanCEO

Prof. Amir B. GevaFounder and CTO

Prof. Leon Y. DeouellFounder and CSO

Sergey VaismanVP R&D

S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 7

S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye

Agenda

▪ Iterative AI Training Framework

▪ Performance

▪ Use Cases

8

Images

Images for human review

Brainwaves Classification Network

Image Classification Network

Trained

Network

InnerEye AI Training Framework

Soft Labels

S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 9

Convergence

Criterion

InnerEye AI Training Framework

Images

Images for human review

Convergence Criterion

Image Classification Network

Trained Network

Deployment

Brainwaves Classification

NetworkSoft Labels

S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye10

Visual Processing

Response Planning

Motor Execution

Distraction caused by

motor response

Sensation

ImageUser

Response

Decision Making

Brain activity measurement

Tapping Into The Brain

InnerEye Bypass

S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 11

Brain Activity Measurement - EEG

t = 0.33s t = 0.66s t = 1s t = 1.33s t = 1.66s t = 2st = 0

S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 12

t = 0.33s t = 0.66s t = 1s t = 1.33s t = 1.66s t = 2st = 0

S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 13

t = 0.33s t = 0.66s t = 1st = 0

Single Trial Spatio-Temporal Activity

time (ms)

Tria

l #

0 200 400 600 800

1000

2000

3000

4000

5000

6000

7000

8000

9000

T Time samplesx D Channels

Single Trial Spatio-Temporal Matrix

S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 14

t = 1st = 0

Single Trial ClassificationTargets

time (ms)

Tria

l #

0 200 400 600 800

100

200

300

400

500

600

Non Targets

time (ms)

Tria

l #

0 200 400 600 800

1000

2000

3000

4000

5000

6000

7000

8000

9000

0 100 200 300 400 500 600 700 800 900-5

0

5

10

time (ms)

mV

PzNon target

Target

t = 200ms t = 300ms t = 450ms t = 550ms t = 700ms t = 200ms t = 300ms t = 450ms t = 550ms t = 700ms

S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 15

InnerEye Brainwaves Classification Network

Deep Neural Network Ensemble

T Time samplesx D Channels

INPUT:Single Trial

Spatio-Temporal Matrix

OUTPUT:Trial

Classification score

S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 16

Brainwaves Classification Scores DistributionScore

Sports CarsRegular Cars

Image

S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 17

Soft Labels Concept▪ These images contain flowers but not all of them should contribute equally to the

learning process

▪ Can we create more informative labels to address the diversity and improve classification accuracy?

Label: FLOWER Label: FLOWER Label: FLOWER

Image Source: Google Open Images Dataset

S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 18

EEG Classification Scores Are Used to Generate Soft Labels

▪ Images that received high or low scores from the EEG classifier are given higher weight

▪ Images that received intermediate (inconclusive) scores from the EEG classifier are given lower weight

𝑤𝑖 = ቊ𝑐𝑖 , 𝑦𝑖 = 1

1 − 𝑐𝑖 , 𝑦𝑖 = 0

𝑐𝑖 = EEG Classification Score

𝑦𝑖 = Sample Label

𝑤𝑖 = Sample Weight (Soft Label)

THR = Classification Threshold

𝑦𝑖 = ቊ1, 𝑐𝑖 ≥ 𝑇𝐻𝑅0, 𝑐𝑖 < 𝑇𝐻𝑅

S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 19

Soft Labels Are Correlated with Human Confidence Level

S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 20

InnerEye AI Training Framework

Images

Images for human review

Convergence Criterion

Image Classification NetworkBrainwaves

Classification Network

Deployment

Soft Labels

S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 21

Image Classification Network▪ In each iteration, image classification layers are trained using the new

generated soft labels64 filters

X 2 blocks 128 filters

X 2 blocks 256 filters

X 3 blocks512 filters

X 3 blocks 512 filters

X 3 blocks 32

neurons

Sigmoid

output

S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 22

Soft Labels Are Used As Sample Weights in Loss Function

▪ We add sample weights to the cross-entropy loss function:

𝐿(𝑥𝑖) = Cross Entropy Loss Function

𝑥𝑖 = Sample

𝑦𝑖 = Sample Label

𝑤𝑖 = Sample Weight (Soft Label)

𝑝𝑖 = Sample Prediction

𝐿 𝑥𝑖 = −𝒘𝒊 𝑦𝑖𝑙𝑜𝑔 𝑝𝑖 + 1 − 𝑦𝑖 𝑙𝑜𝑔(1 − 𝑝𝑖)

S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 23

▪ The learning algorithm selects new samples to learn from in the next iteration based on Confidence criterion:

▪ Only the least confident samples (𝐶𝑖 < THR) will be sent for human review

▪ Also used as Convergence Condition

Convergence and Active Learning Iterations

𝐶𝑖 = max𝑗

𝑃 𝑦𝑖 = 𝑗|𝑥𝑖

𝐶𝑖 = Confidence for Sample i

j = Class index

𝑦𝑖 = Predicted Sample label

𝑥𝑖 = Sample i

S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 24

Soft Labels Improve Neural Network Performance

(*) AUC shown after the first iteration

(permuted sample weights of the labels)

S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 25

Active Learning Combined With Soft Labels Improves Neural Network Performance

(*) AUC shown after the first iteration

S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 26

Images

Images for human review

Brainwaves Classification Network

Image Classification Network

Trained

Network

InnerEye AI Training Framework

Soft Labels

S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye

Convergence

Criterion

27

S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye

Agenda

▪ Iterative AI Training Framework

▪ Performance

▪ Use Cases

28

We started with 2,800 Cars images to be classified. Initial

network performance for Sports cars detection: AUC=0.5

S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 29

After Iteration 1 (T=4.2 min): Human expert reviewed

200 images. Network performance: AUC=0.8

S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 30

After Iteration 2 (T=8.7 min): Human expert reviewed

400 images. Network performance: AUC=0.86

S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 31

After Iteration 3 (T=13.3 min): Human expert reviewed

589 images. Network performance: AUC=0.88

S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 32

After Iteration 4 (T=17.2 min): Human expert reviewed

718 images. Network performance: AUC=0.9

S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 33

After Iteration 5 (T=20.6 min): Human expert reviewed

795 images. Network performance: AUC=0.92

S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 34

After 20.6 minutes the network is trained to detect Sports Cars.

The human expert was required to review only 28% of the images

S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 35

Examples of correctly classified sports cars:Images with the highest score from the InnerEye system trained to classify Sport Cars:

S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 36

S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 37

Image Classification Performance vs. % of Training Data

38

(*) AUC shown after the first iteration

Subject 1 Subject 2 Subject 3

S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 38

Image Classification Performance vs. Training Time

(*) Trained on NVIDIA GeForce GTX 1080 Ti

0.5

0.55

0.6

0.65

0.7

0.75

0.8

0.85

0.9

0.95

0 25 50 75 100 125 150 175 200 225 250 275

AU

C

Time [min]

Subject 1

InnerEye Training Method

Manual Training Method

0.5

0.55

0.6

0.65

0.7

0.75

0.8

0.85

0.9

0.95

0 25 50 75 100 125 150 175 200 225 250 275

AU

C

Time [min]

Subject 2

InnerEye Training Method

Manual Training Method

0.5

0.55

0.6

0.65

0.7

0.75

0.8

0.85

0.9

0.95

0 25 50 75 100 125 150 175 200 225 250 275

AU

C

Time [min]

Subject 3

InnerEye Training Method

Manual Training Method

▪ Time savings come from combination of fast presentation rate and faster convergence

S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 39

Image Classification Average Performance

10x times faster

Trained on NVIDIA GeForce GTX1080 ti GPU

S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 40

Soft Labels Compensate for EEG Misclassified Samples Correctly classified sports cars

Weight: 0.93 Weight: 0.81 Weight: 0.87

False Positives

Weight: 0.26Weight: 0.3 Weight:0.22

Weight: 0.4 Weight: 0.43 Weight: 0.31

Missed sports cars

Weight: 0.85 Weight: 0.79 Weight: 0.91

Correctly classified regular cars

S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 41

Soft Labels Compensate for EEG Misclassified Samples

(*) AUC shown after the first iteration

S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 42

S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye

Agenda

▪ Iterative AI Training Framework

▪ Performance

▪ Use Cases

43

Multiclass ClassificationAccuracy: 0.81

Event Related Potentials measured on electrode T5

S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 44

Validation of Annotation Quality

▪Fast Screening and amendment of low confidence annotated data

Screenshot of the output from InnerEye system output of detecting images wrongly labeled as “flowers”

Source: Google Open Images Dataset

S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 45

Combined Brain-Computer Visual Network

IMG Net

EEG Net

Score: Classification:

S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 46

InnerEye Driver

Car SensorsRoad Conditions

Environment:

Driving Agent

State Reward Action

Screenshot of the output from “InnerEye driver” brain responding to seeing pedestrian crossing the road, measuring Hazard Detection, Attention and Emotion

Reinforcement Learning for Autonomous Driving▪ Incorporating brain insights in the training process of the AI driving agent

S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 47

Efficiency & Throughput

Cost

PersonalizationReal-Time Interface

Privacy

Summary

S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 48

THANK YOU!COME SEE OUR DEMO

AT BOOTH 335

Contact me at: sergey@innereye.ai

S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 49