DEEP LEARNING IN THE CLOUD: TENSORFLOW VS MXNET

ASHISH BANSAL, SNR DIR DATA SCIENCE @ CAPITAL ONE

@ASH_BANS | LINKEDIN.COM/IN/BANSALASHISH

AGENDA FOR TODAY

• What is deep learning

• Pre-requisites for deep learning

• Libraries landscape

• TensorFlow vs MxNet

• Programming model

• Popularity, developer support, cloud, deployment

• Summary

Deep Learning in the cloud Ashish Bansal 1

WHAT IS DEEP LEARNING?• Variant of neural networks

• Inspired by human brain, but only very loosely mimics it (very very simple compared to the brain)

• Saves a ton of data scientist time in feature engineering–spend more time in tuning

• GPUs make it very efficient. Thank you Gamers!

• Requires large amounts of data

• Given enough data, will generally outperform other algorithms

• All popular references to AI that you read refer to deep learning advances

Deep Learning in the cloud Ashish Bansal 2

WHAT MAKES DEEP LEARNING DEEP: FEATURE HIERARCHY OF TRAINED REPRESENTATIONS

Deep Learning in the cloud Ashish Bansal 3

• 2 layer models are not deep because there is no feature

hierarchy

• Neural networks with 1 hidden layer are not considered

deep for the same reason

• Classification trees are not deep as all decisions are

made in the input space

Source: Unreasonable Effectiveness of Deep Learning by Yann LeCun (http://www4.ncsu.edu/~pcombet/data2015/lecun2015.pdf)

PRE-REQUISITES FOR DEEP LEARNING

• Data, and lots of it

• A way to express or code neural networks

• A fast processor adept at matrix computations

• High throughput between storage and compute for several passes during

training

Note: Training, by far, is the most time taking activity. Once trained, networks can classify fairly quickly

Deep Learning in the cloud Ashish Bansal 4

PRE-REQUISITES FOR DEEP LEARNING

• Data, and lots of it: Is it on the cloud? Key challenge here is getting data to the cloud

• A way to express or code neural networks: Can be solved with TensorFlow or MxNet

• A fast processor adept at matrix computations: GPU based cloud units like AWS P2

instances

• High throughput between storage and compute for several passes during training:

Large memory on the cloud instances, like p2.8xlarge, and parallelization

Deep Learning in the cloud Ashish Bansal 5

LIBRARIES LANDSCAPE

Deep Learning in the cloud Ashish Bansal 6

TENSORFLOW & MXNET OVERVIEW

Deep Learning in the cloud Ashish Bansal Source: https://svds.com/getting-started-deep-learning/ 7

TENSORFLOW VS MXNET: GITHUB VIEW

Deep Learning in the cloud Ashish Bansal Source: Github. Data as of Apr 18, 2017

8

0

10000

20000

30000

40000

50000

60000

Stars Forks Commits Contributors

Deep Learning Libraries: Github Metrics

TensorFlow Keras MxNet Qix Torch Caffe Theano DL4J CNTK DSSTNE

TENSORFLOW VS MXNET: STACK OVERFLOW VIEW

Deep Learning in the cloud Ashish Bansal Source: Stack Overflow, Data as of Apr 18, 2017 9

0

5000

10000

15000

20000

25000

TensorFlow Keras MxNet Qix Torch Caffe Theano DL4J CNTK DSSTNE

Stack Oveflow Activity

TENSORFLOW VS MXNET: PERFORMANCE VIEW

Deep Learning in the cloud Ashish Bansal Source: https://arxiv.org/pdf/1608.07249.pdf 10

AlexNet Performance (5 layers, 81K params) ResNet-56 Performance (56 layers, 850K params)

TENSORFLOW VS MXNET: CLOUD SUPPORT VIEW

• TensorFlow is fully supported on AWS EC2, Google Cloud & Microsoft Azure

• MxNet is fully supported on AWS EC2 (preferred) and Microsoft Azure

• Your mileage may vary for MxNet on Google Cloud

Deep Learning in the cloud Ashish Bansal 11

TENSORFLOW VS MXNET: DEPLOYMENT VIEW

TensorFlow MxNet

Android Yes OOB Yes – needs additional libs & compilation

iOS Yes OOB Yes – needs additional libs & compilation

Raspberry Pi Yes OOB Yes OOB

Windows YMMV Yes

Unix/Linux Yes OOB Yes OOB

Deep Learning in the cloud Ashish Bansal 12

SUMMARY

• Lots of momentum and support behind TensorFlow

• TensorFlow has better RNN capabilities

• TensorFlow has better tutorials and online guides. It also has more

supporting material like Stack Overflow questions etc.

• TensorFlow enjoys greater support on the cloud, and has more

deployment options

• MxNet has more language bindings, and is usually faster

Deep Learning in the cloud Ashish Bansal 13

QUESTIONS@ASH_BANS | LINKEDIN.COM/IN/BANSALASHISH

Deep Learning in the cloud Ashish Bansal 14