View
212
Download
1
Category
Preview:
Citation preview
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.Webinars
Sunil Mallya
Solutions Architect, Deep Learning
A Deeper Dive into Apache MXNet on AWS
Agenda
• Apache MXNet introduction• Distributed Deep Learning with AWS Cloudformation• Deep Learning motivation and basics• MXNet programing model overview• Train our first neural network using MXNet
Deep Learning ApplicationsSignificantly improve many applications on multiple domains
image understanding speech recognition natural language processing
autonomy
• Netflix – Recommendation Engine• FINRA – Anonmaly detection, Sequence matching• TuSimple - Computer Vision for Autonomous Driving• Pinterest - Image recognition search• Mapillary - Computer vision for crowd sourced maps
AI Customers on AWS
AI Services
AI Platform
AI Engines
Amazon Rekognition
Amazon Polly
Amazon Lex
More to comein 2017
Amazon Machine Learning
Amazon Elastic MapReduce
Spark & SparkML
More to comein 2017
Apache MXNet TensorFlow Caffe Theano KerasTorch CNTK
P2 ECS LambdaEMR/Spark GreenGrass FPGA More to comein 2017
Hardware
Democratizing Artificial Intelligence
Apache MXNet
Programmable Portable High PerformanceNear linear scaling
across hundreds of GPUsHighly efficient
models for mobileand IoT
Simple syntax, multiple languages
88% efficiencyon 256 GPUs
Resnet 1024 layer network is ~4GB
Webinars
Distributed Deep Learning
IdealInception v3Resnet
Alexnet
88%Efficiency
1 2 4 8 16 32 64 128 256No. of GPUs
• Cloud formation with Deep Learning AMI
• 16x P2.16xlarge. Mounted on EFS
• Inception and Resnet: batch size 32, Alex net: batch size 512
• ImageNet, 1.2M images,1K classes
• 152-layer ResNet, 5.4d on 4x K80s (1.2h per epoch), 0.22 top-1 error
Scaling with MXNet
Distributed Training Setup with Cloudformation
https://github.com/awslabs/deeplearning-cfn
Webinars
Deep Learning basics
Biological Neuron
slide from http://cs231n.stanford.edu/
Artificial Neuron
output
synapticweights
• InputVector of training data x
• OutputLinear function of inputs
• NonlinearityTransform output into desired range of values, e.g. for classification we need probabilities [0, 1]
• TrainingLearn the weights w and bias b
Deep Neural Network
hidden layers
The optimal size of the hidden layer (number of neurons) is usually between the size of the input and size of the output layers
Input layer
output
The “Learning” in Deep Learning
0.4 0.3
0.2 0.9
...
back propogation (gradient descent)
X1 != X0.4 ± 𝛿 0.3 ± 𝛿
newweights
newweights
01011
.
.--X
input
label
...X1
Hidden Layer Visualization
Webinars
MXNet Programing Model
import numpy as npa = np.ones(10)b = np.ones(10) * 2c = b * a
• Straightforward and flexible.• Take advantage of language
native features (loop, condition, debugger)
• E.g. Numpy, Matlab, Torch, …
• Hard to optimize
PROS
CONSd = c + 1c
Easy to tweak with python codes
Imperative Programing
• More chances for optimization• Cross different languages• E.g. TensorFlow, Theano,
Caffe
• Less flexible
PROS
CONSC can share memory with D because C is deleted later
A = Variable('A')B = Variable('B')C = B * AD = C + 1f = compile(D)d = f(A=np.ones(10),
B=np.ones(10)*2)
A B
1
+
X
Declarative Programing
IMPERATIVE NDARRAY API
DECLARATIVE SYMBOLIC EXECUTOR
>>> import mxnet as mx>>> a = mx.nd.zeros((100, 50))>>> b = mx.nd.ones((100, 50))>>> c = a + b>>> c += 1>>> print(c)
>>> import mxnet as mx>>> net = mx.symbol.Variable('data')>>> net = mx.symbol.FullyConnected(data=net, num_hidden=128)>>> net = mx.symbol.SoftmaxOutput(data=net)>>> texec = mx.module.Module(net)>>> texec.forward(data=c)>>> texec.backward() NDArray can be set
as input to the graph
MXNet: Mixed programming paradigm
Webinars
Lets train our first model to classify handwritten digits
MXNet Overview
• Founded by: U.Washington, Carnegie Mellon U. (~1.5yrs old)• Recently Accepted to the Apache Incubator • State of the Art Model Support: Convolutional Neural Networks (CNN), Long
Short-Term Memory (LSTM)• Scalable: Near-linear scaling equals fastest time to model• Multi-language: Support for Scala, Python, R, etc.. for legacy code leverage and
easy integration with Spark• Ecosystem: Vibrant community from Academia and Industry
Open Source Project on Github | Apache-2 Licensed
Application Examples | Python notebooks• https://github.com/dmlc/mxnet-notebooks• Basic concepts
• NDArray - multi-dimensional array computation• Symbol - symbolic expression for neural networks• Module - neural network training and inference
• Applications• MNIST: recognize handwritten digits• Check out the distributed training results• Predict with pre-trained models• LSTMs for sequence learning• Recommender systems• Train a state of the art Computer Vision model (CNN)• Lots more..
Call to ActionMXNet Resources:• MXNet Blog Post | AWS Endorsement • Read up on MXNet and Learn More: mxnet.io• MXNet Github Repo • MXNet Recommender Systems Talk | Leo DiracDeveloper Resources:• Deep Learning AMI | Amazon Linux• Deep Learning AMI | Ubuntu – NEW!!!• P2 Instance Information• CloudFormation Template Instructions• Deep Learning Benchmark • MXNet on Lambda • MXNet on ECS/Docker• MXNet on Raspberry Pi | Wine Detector
Webinars
Thank You
smallya@amazon.comsunilmallya
Recommended