Efficient and Scalable Deep LearningWeiWen, Yiran Chen, Hai(Helen)Li, DukeUniversity

{wei.wen,yiran.chen,hai.li}@duke.edu

Efficient Deep Learning for Faster Inference on the Edge and in the Cloud

(Alfredo et al. 2016)60

62

64

66

68

70

72

74

76

0 10 20 30 40 50 60

TestPerplexity vsModel Size

VDLSTM,Inan etal.(2016)

NAS,Zoph &Le(2016)

RHN,Zilly etal.(2016)

ImageClassificationonImageNet LanguageModelingonPennTreebank

M

More accurate predictionbypayingcomputation cost using larger models.

Inference on the edge:• Limited computing capability• Limited memory• Limited batteryenergy

Wired PopularMechanics

A machine:48CPUs and8GPUs; a distributedversion: 1,202CPUsand176GPUs (Silver et al. 2016)

“Prototypesusearound2,500watts”

Inference in the cloud:• A challenge: Real-time response to

trillions of AI service requests• Facebook (K. Hazelwood@ISCA’18)

• 200+ Trillion predictions per day• 5+ Billion language translations per dayWe must compress and accelerate deep learning models.

Wired

OSXDaily

Fortune

IBMCloudAI

One of Our solutions: Structurally Sparse Deep Neural Networks (SSDNNs)

Group Lasso Regularization is All you Need for SSDNNs

!!!argmin

wE w( ){ }= argmin

wED w( )+λg ⋅Rg w( ){ }

Step 1: WeightsaresplittoG groups w(1..G)

i.e. vector lengthStep 2: Add group Lasso on each group w(g)

Step 3: Sum group Lasso over all groups as a regularization:

Step 4: SGD optimizing

e.g. (w1, w2, w3,w4, w5) -> group 1: (w1, w2, w3), group 2: (w4, w5)

sqrt(w12 + w2

2 +w32), sqrt(w4

2 + w52)

sqrt(w12 + w2

2 +w32) + sqrt(w4

2 + w52)

We refer to our method as Structured Sparsity Learning (SSL)

Structurally Sparse AlexNet

0

20

40

60

80

100

0

20

40

60

80

100

41.5 42 42.5 43 43.5 44

conv1 conv2 conv3 conv4 conv5 FLOP

% Sparsity % FLOP reduction

% top-1 error

00.10.20.30.40.50.60.70.80.91

0123456789

10

1 2 3 4 5

AlexNet: sparsity and speedup

Intel Xeonspeedup

Nvidia TitanBlack speedup

Speedup Sparsity

Sparsity

conv

Removing Columns and Rows: Removing 2D convolution:

Structurally Sparse LSTMs

LSTM 1 LSTM 2 Output

Learned structurallysparse LSTMs

ours

SSL is able to remove: filters,channels, neurons, layers,matrix dimensions, hiddenstates, weight blocks, etc.

Our works on Efficient Deep Learning: NIPS 2016, ICLR 2018, ICCV 2017, ICLR 2017, CVPR 2017, ASP-DAC 2017 (Best Paper Award), DAC 2015 & 2016 (Two Best Paper Nominations).

Scalable Deep Learning for Faster Training in Distributed Systems

Challenges of Training Deep Learning Models

Google TPU CloudRiseML

Jeffreyetal.2

012

(K. Hazelwood@ISCA’18) Picture: leiphone

parallel

sequential(bottleneck)

# machines

Time

Total Limitation

(1) Communication bottleneck

Priya et al. 2017

ResNet-50

(2) Bad generalizationin large-batchSGD

Gradient Quantization for Communication Reduction

ternary gradients (TernGrad)

Parameter server!" = $!"

(&)�

�

Worker 1)"*+ ← )" − ."!"

Worker 2)"*+ ← )" −."!"

Worker N)"*+ ← )" − ."!"

……!"(+)

!"(/)

!"(0)

!" !" !"

(a) original (b) clipped (c) ternary (d) final

Iteration#

Iteration#

conv

fc

< 2 bits (>16x reduction)

It’s challenging because it losesprecision in the learning whichrelies on small updates

g̃t = ternarize(gt) = st · sign (gt) � btst , ||gt||1 , max (abs (gt))⇢P (btk = 1 | gt) = |gtk|/stP (btk = 0 | gt) = 1� |gtk|/st

An Example:𝒈"($):[0.30,-1.20,…,0.9]

𝑠":1.20Signs: [1, -1, …, 1]𝑃 𝑏") = 1|𝑔" :[..0

1.2, 1.21.2,…,..4

1.2]

𝒃":[0,1,…,1]𝒈"($)7 :[0,-1,…,1]*1.20

TernGrad: quantizing gradients to reduce communication (NIPS 2017, Oral)

(a) (b)

0

20000

40000

60000

80000

100000

Imag

es/s

ec

# of GPUs

Training throughput on GPU cluster with Ethernet and PCI switch

AlexNet FP32 AlexNet TernGradGoogLeNet FP32 GoogLeNet TernGradVggNet-A FP32 VggNet-A TernGrad

1 2 4 8 16 32 64 128 256 512 0

40000

80000

120000

160000

200000

240000

Imag

es/s

ec

# of GPUs

Training throughput on GPU cluster with InfiniBand and NVLink

AlexNet FP32 AlexNet TernGradGoogLeNet FP32 GoogLeNet TernGradVggNet-A FP32 VggNet-A TernGrad

1 2 4 8 16 32 64 128 256 512

0

1000

2000

3000

4000

0

2000

4000

6000

0%10%20%30%40%50%60%70%

0 50000 100000 150000

baselineterngrad

0

2

4

6

8

0 50000 100000 150000

baselineterngrad

0%

20%

40%

60%

80%

0 50000 100000 150000

(c) gradient sparsity of terngrad in fc6(b) training loss vs iteration(a) top-1 accuracy vs iteration

TernGrad Summary:ü No loss in AlexNet;ü <2% loss in GoogLeNet

TernGrad is now in PyTorch/Caffe2 and Adopted in Facebook AI Infra.

W. Wen, Y. Wang, F.Yan, C. Xu, Y. Chen, H. Li, “SmoothOut: Smoothing Out Sharp Minima to Improve Generalization in Deep Learning”, AAAI 2018Submission.

Solution to (1)

Solution to (2)

Ref.Ref.

Ref.

Applied to:

(Convergence guaranteed)

Bad

Good