COMS0018: PRACTICAL-Lab4 · 2019-11-11 · COMSM0018: PRACTICAL - Lab4 - 2018/2019. Debugging Deep...

COMS0018: PRACTICAL-Lab4

Dima DamenDima.Damen@bristol.ac.uk

Bristol University, Department of Computer ScienceBristol BS8 1UB, UK

October 26, 2018

Dima DamenDima.Damen@bristol.ac.uk

COMSM0018: PRACTICAL - Lab4 - 2018/2019

What else can we do?

I Data AugmentationI Assessing Dataset SizeI Debugging Strategies

Dima DamenDima.Damen@bristol.ac.uk

COMSM0018: PRACTICAL - Lab4 - 2018/2019

Data Augmentation

I Data augmentation is making the most of the training samples byintroducing variations of these samples to accommodate for requiredinvariances

I Why Data Augmentation?I Because it’s all about the size of your data –> More data for trainingI More importantly... to accommodate invariances

Dima DamenDima.Damen@bristol.ac.uk

COMSM0018: PRACTICAL - Lab4 - 2018/2019

Invariances in data

I A problem is invariant to a property when the problem remainsunchanged when transformations of a certain type are applied.

Problem Invariant to...Object Recognition translation, rotation, scaling,viewpointNumber plate recognition translation, scalingAction Recognition translation, rotation, scaling, viewpoint, speed

Dima DamenDima.Damen@bristol.ac.uk

COMSM0018: PRACTICAL - Lab4 - 2018/2019

Invariances in data

Problem Invariant to...Object Recognition translation, rotation, scaling, viewpoint

I How to provide invariance? → artificially augment for:I Translation: shifts – luckily CNNs do that for usI Rotation: rotationsI Scaling: croppingsI Viewpoint: Minor - affine transformations, otherwise :-( collect more data!

Dima DamenDima.Damen@bristol.ac.uk

COMSM0018: PRACTICAL - Lab4 - 2018/2019

Invariances in data

I Other invariances:I invariance to random noiseI invariance to occlusionI invariance to lighting conditionsI invariance to time of year!? — Generative!

Dima DamenDima.Damen@bristol.ac.uk

COMSM0018: PRACTICAL - Lab4 - 2018/2019

Data Augmentation

I Data augmentation for invariances existed before deep learning

Ozuysal et al (2010). Fast Keypoint Recognition Using Random Ferns. TPAMI.

Dima DamenDima.Damen@bristol.ac.uk

COMSM0018: PRACTICAL - Lab4 - 2018/2019

Data Augmentation

I Why can’t current deep learning methods do that automatically for us?

Dima DamenDima.Damen@bristol.ac.uk

COMSM0018: PRACTICAL - Lab4 - 2018/2019

But... Do I need more Data?

I There is a balance between the expense of collecting labelled dataand refining the method

I How are you performing on your training data?I poorly → your algorithm needs work, you are not making the most of the

data you haveI quite well → How are you performing on your test data?

I poorly → there is little you can do, collect more dataI quite well → you’re done! Find a more interesting problem

I I collected more data but things did not improve?I Think about the quality of your dataI Think about the quality of your labelsI → fix and start over

Dima DamenDima.Damen@bristol.ac.uk

COMSM0018: PRACTICAL - Lab4 - 2018/2019

Debugging Deep Learning Algorithms

I When a general machine learning code performs poorly, includingdeep learning code, it is very tricky to decide whether that is a bug inthe code or a problem in the algorithm

I Compiling correctly and getting numbers out is not an indication ofcorrectness

I We do not know what the "correct" implementation will give in terms ofaccuracy, that is in fact what we wish to discover

I Careful debugging is thus a must

Dima DamenDima.Damen@bristol.ac.uk

COMSM0018: PRACTICAL - Lab4 - 2018/2019

What is your baseline performance?

I Remember: you cannot perform worse than the baseline!I What can an algorithm that makes decisions by “chance” do?I For a binary classifier, your baseline is 50%I For a classifier into N balanced classes, your baseline is 1

N%

I For a classifier with unbalanced classes??

Dima DamenDima.Damen@bristol.ac.uk

COMSM0018: PRACTICAL - Lab4 - 2018/2019

Ex. of Baselines - EPIC-Kitchens 2018

I https://epic-kitchens.github.io

I https://youtu.be/Dj6Y3H0ubDw

Dima DamenDima.Damen@bristol.ac.uk

COMSM0018: PRACTICAL - Lab4 - 2018/2019

Ex. of Baselines - EPIC-Kitchens 2018

D Damen, H Doughty, GM Farinella, S Fidler, A Furnari, E Kazakos, D Moltisanti, J Munro, T Perrett, W Price, M Wray (2018). Scaling EgocentricVision: The EPIC-KITCHENS Dataset. European Conference on Computer Vision (ECCV)

Dima DamenDima.Damen@bristol.ac.uk

COMSM0018: PRACTICAL - Lab4 - 2018/2019

Ex. of Baselines - EPIC-Kitchens 2018

D Damen, H Doughty, GM Farinella, S Fidler, A Furnari, E Kazakos, D Moltisanti, J Munro, T Perrett, W Price, M Wray (2018). Scaling EgocentricVision: The EPIC-KITCHENS Dataset. European Conference on Computer Vision (ECCV)

Dima DamenDima.Damen@bristol.ac.uk

COMSM0018: PRACTICAL - Lab4 - 2018/2019

Debugging Strategies

I Mickey Mouse Examples - test your solution on small tests that youknow the outcome for

I Evaluate the performance of your building blocks in isolationI Monitor the model in actionI Look at failure cases (qualitative assessment)

Dima DamenDima.Damen@bristol.ac.uk

COMSM0018: PRACTICAL - Lab4 - 2018/2019

Debugging Strategies

“Directly observing the machine learning model performing its task willhelp to determine whether the quantitative performance numbers itachieves seem reasonable”

Goodfellow et al, p432

Dima DamenDima.Damen@bristol.ac.uk

COMSM0018: PRACTICAL - Lab4 - 2018/2019

Debugging Strategies

“Evaluation bugs can be some of the most devastating bugs because theycan mislead you into believing your system is performing well when it isnot”

Goodfellow et al, p432

Dima DamenDima.Damen@bristol.ac.uk

COMSM0018: PRACTICAL - Lab4 - 2018/2019

And now....

READY....

STEADY....

GO...

Dima DamenDima.Damen@bristol.ac.uk

COMSM0018: PRACTICAL - Lab4 - 2018/2019