Introduction to Deep Learning with Tensor owIntroduction to Deep Learning with Tensor ow Yunhao (Robin) Tang Department of IEOR, Columbia University February 25, 2019 IEOR 8100: Reinforcement

PreliminaryIntro to DL with Tensorflow

Introduction to Deep Learning with Tensorflow

Yunhao (Robin) TangDepartment of IEOR, Columbia University

February 25, 2019

IEOR 8100: Reinforcement Learning 1/36


Summary

Summary

What we will cover...

Simple Example: Linear Regression with Tensorflow

Intro to Deep Learning

Deep Learning with Tensorflow

Beyond

If time permits, we will also cover

OpenAI Gym

Implement basic DQN



Summary

Summary





Beyond


OpenAI Gym

Implement basic DQN



Summary

Summary





Beyond


OpenAI Gym

Implement basic DQN



Summary

Summary





Beyond


OpenAI Gym

Implement basic DQN



Summary

Summary





Beyond


OpenAI Gym

Implement basic DQN



Summary

Summary





Beyond


OpenAI Gym

Implement basic DQN



Summary

Summary





Beyond


OpenAI Gym

Implement basic DQN



Simple ExampleIntro to DLBeyond Tensorflow

Info about Tensorflow

TF is an open source project formally supported by Google.

Programming interface: Python, C++, C... Python is most well developed anddocumented.

Latest version: 1.12 (same time last year 1.5). Commonly used version: ≈1.0.(depend on other dependencies)

Can be installed via pip. Best supported on mac os and linux.

(a) Tensorflow paper









(b) Tensorflow paper









(c) Tensorflow paper









(d) Tensorflow paper









(e) Tensorflow paper




Simple Example: Linear Regression

Linear Regression: N data points {xi , yi}Ni=1, x ∈ Rk , y ∈ RSpecify Architecture: Consider linear function for regression with parameter:slope θ ∈ Rk and bias θ0 ∈ R. The prediction is

yi = θT xi + θ0

Define Loss: Loss function to minimize J = 1N

∑Ni=1(yi − yi )

2

Gradient Descent: θ ← θ − α∇θJ, θ0 ← θ0 − α∇θ0J





Linear Regression: N data points {xi , yi}Ni=1, x ∈ Rk , y ∈ R

Specify Architecture: Consider linear function for regression with parameter:slope θ ∈ Rk and bias θ0 ∈ R. The prediction is

yi = θT xi + θ0


∑Ni=1(yi − yi )

2







yi = θT xi + θ0


∑Ni=1(yi − yi )

2







yi = θT xi + θ0


∑Ni=1(yi − yi )

2







yi = θT xi + θ0


∑Ni=1(yi − yi )

2





Linear Regression: Data

(f) True Parameters

(g) Data




Linear Regression with Tensorflow

Let us just focus on defining architecture...

(h) Linear regression tf code




Linear Regression with Tensorflow

We launch the training...

(i) loss (j) fitted line




Intro to DL

Consider data sets with complex relation {xi , yi}Ni=1, linear regression will notwork.MLP (Multi Layer Perceptron): stack multiple layers of linear transformationsand nonlinear functions.

input x ∈ R6, first layer linear transformation W1x + b1 ∈ R4, then apply nonlinearfunction h1 = σ(W1x + b1) ∈ R4. Nonlinear function applies elementwise.second layer has W2, b2, transform as h2 = σ(W2h2 + b2) ∈ R3

final output y = W3σ(W2σ(W1x + b1) + b2) + b3 ∈ R

(k) mlp (l) nonlinear functions




Intro to DL

Consider data sets with complex relation {xi , yi}Ni=1, linear regression will notwork.

MLP (Multi Layer Perceptron): stack multiple layers of linear transformationsand nonlinear functions.



(m) mlp (n) nonlinear functions




Intro to DL




(o) mlp (p) nonlinear functions




Intro to DL


input x ∈ R6, first layer linear transformation W1x + b1 ∈ R4, then apply nonlinearfunction h1 = σ(W1x + b1) ∈ R4. Nonlinear function applies elementwise.

second layer has W2, b2, transform as h2 = σ(W2h2 + b2) ∈ R3


(q) mlp (r) nonlinear functions




Intro to DL




(s) mlp (t) nonlinear functions




Intro to DL




(u) mlp (v) nonlinear functions




Intro to DL

Architecture: define a complex input to output relation with stacked layers oflinear transformations and nonlinear functions (activations).

Architectures are adapted to data structure at hand.Input prior knowledge into modeling through architecture design.MLP, CNN (image), RNN (sequence data, audio, language...

Parameters: weights W1,W2,W3 and bias b1, b2, b3.Linear regression: slope θ and bias θ0




Intro to DL







Intro to DL


Architectures are adapted to data structure at hand.

Input prior knowledge into modeling through architecture design.MLP, CNN (image), RNN (sequence data, audio, language...





Intro to DL


Architectures are adapted to data structure at hand.Input prior knowledge into modeling through architecture design.

MLP, CNN (image), RNN (sequence data, audio, language...





Intro to DL







Intro to DL



Parameters: weights W1,W2,W3 and bias b1, b2, b3.

Linear regression: slope θ and bias θ0




Intro to DL







DL algorithm

Specify Architecture: how many layers, how many hidden unit per layer?yi = fθ(xi ) with generic parameters θ (weights and biases).


∑Ni=1(yi − yi )

2


Algorithm 1 Generic DL regression

1: Input: model architecture, data {xi , yi}, learning rate α2: Initialize: Model parameters θ = {Wi , bi}3: for t=1,2,3...T do4: Compute prediction yi = fθ(xi )5: Compute loss J = 1

N

∑i (yi − yi )

2

6: Gradient update θ ← θ − α∇θJ7: end for




DL algorithm



∑Ni=1(yi − yi )

2




N

∑i (yi − yi )

2





DL algorithm



∑Ni=1(yi − yi )

2




N

∑i (yi − yi )

2





DL algorithm



∑Ni=1(yi − yi )

2




N

∑i (yi − yi )

2





DL Regression: Data

Generate data using y = x3 + x2 + sin(10x) + noise. Linear regression will fail.

(w) data generation

(x) data plots




DL Regression with Tensorflow

Specify the Architecture, Loss using Tensorflow syntax just like linear regression.

(y) Linear regression tf code

train the model just like linear regression





Specify the Architecture, Loss using Tensorflow syntax just like linear regression.

(z) Linear regression tf code

train the model just like linear regression





We launch the training...

() loss () fitted line




Batch training

When data set is large, can only compute gradients on mini-batches.

Sample a batch of data → Compute gradient on the batch → Update parameters.

Batch size: too small: high variance in data stream, unstable training; too large:too costly to compute gradients.




Batch training







Batch training







Batch training







Auto-differentiation

Taking gradients in linear model is easy.

Question: how to take gradients for MLP? ∇W1J is not straightforward as inlinear regression.

J =1

N

∑i

(yi − yi )2, yi = W3σ(W2σ(W1xi + b1) + b2) + b3







J =1

N

∑i








J =1

N

∑i






Tensorflow entails automatic differentiation (autodiff), i.e. automatic gradientcomputation. Users specify forward computation y = fθ(x), the program willinternally specify a way to compute ∇θy . Not symbolic computation, not finitedifference

Example: start with a, b, compute e: c = a + b, d = b + 1, e = c · d . Consider

∂e

∂a=∂e

∂d

∂d

∂a+∂e

∂c

∂c

∂a

() computation graph







∂e

∂a=∂e

∂d

∂d

∂a+∂e

∂c

∂c

∂a








∂e

∂a=∂e

∂d

∂d

∂a+∂e

∂c

∂c

∂a





DL with Tensorflow

High level idea: computation graph. Forward computation specifies localoperations that connect locally related variables. Local gradients can becomputed.

forward computation → local forward ops → local gradient → long rangegradients.

Just like error back-propagating from output to inputs and params,back-propagation.

Useful link: http://deeplearning.stanford.edu/wiki/index.php/UFLDL Tutorial




DL with Tensorflow








DL with Tensorflow








DL with Tensorflow








DL with Tensorflow








DL with Tensorflow

What we have covered from the code...

Placeholder: tf objects that specify users’ inputs into the graph, as literallyplaceholders for data.

Variable: tf objects that represent parameters that can be updated throughautodiff. Embedded inside the graph.

All expressions computed from primitive variables and placeholders are vertices inthe graph.




DL with Tensorflow








DL with Tensorflow








DL with Tensorflow








DL with Tensorflow

What we have not yet covered from the code... how to launch the training? how touse the gradients computed by Tensorflow to update parameters?

TF expression: python objects that define how to update the graph (variables inthe graph) in a single step.

Optimizer: python objects that can specify gradient updates of variables.

Session: python objects that launch the training.

And many more...




DL with Tensorflow





And many more...




DL with Tensorflow





And many more...




DL with Tensorflow





And many more...




DL with Tensorflow





And many more...




Launch the training

Define loss and optimizers: which specifies how to update model parametersduring training.

Define session to launch the training

Initialize variables

Feed data into the computation graph

() launch the training




Launch the training









Launch the training









Launch the training









Launch the training









Training: whole landscape





Beyond Regression

Classification: regression on probability (image classification)

Structured Prediction: regression on high dimensional objects (autoencoders)

Reinforcement Learning: DQN, policy gradients...DQN is an analogy to regressionPG is an analogy to classification




Beyond Regression







Beyond Regression







Beyond Regression



Reinforcement Learning: DQN, policy gradients...

DQN is an analogy to regressionPG is an analogy to classification




Beyond Regression



Reinforcement Learning: DQN, policy gradients...DQN is an analogy to regression

PG is an analogy to classification




Beyond Regression







Hyper-parameters

Get a DL model to work involves a lot of hyper-parameters.

Optimization: optimization subroutines that update parameters with gradients.beyond simple sgd, there are rmsprop, adam, adagrad... adam is popular.learning rate

Architectures: number of layers, hidden units per layers

Nonlinear function: sigmoid, relu, tanh...

Initialization: initialization of variables W , b.arbitrary initializations will fail.xavier initialization, truncated normal...

Need hand tuning or cross validation to select good hyper-parameters.

Other topics: batch-normalization, dropout, stochastic layers...




Hyper-parameters


Optimization: optimization subroutines that update parameters with gradients.

beyond simple sgd, there are rmsprop, adam, adagrad... adam is popular.learning rate









Hyper-parameters


Optimization: optimization subroutines that update parameters with gradients.beyond simple sgd, there are rmsprop, adam, adagrad... adam is popular.

learning rate









Hyper-parameters











Hyper-parameters











Hyper-parameters











Hyper-parameters





Initialization: initialization of variables W , b.

arbitrary initializations will fail.xavier initialization, truncated normal...






Hyper-parameters





Initialization: initialization of variables W , b.arbitrary initializations will fail.

xavier initialization, truncated normal...






Hyper-parameters











Hyper-parameters











Hyper-parameters











Beyond Tensorflow

Other autodiff softwares: Pytorch/Chainer/Theano/Caffepytorch and chainer allow for dynamic graph buildingothers use static graph buildingstrengths/weaknesses: gpu, distributed computing, model building flexibility,debug...

High level interface: Kerasuse tf and theano as backendspecify model architecture more easily

High level interface: stick with Tensorflowtensorflow.contrib




Beyond Tensorflow

Other autodiff softwares: Pytorch/Chainer/Theano/Caffe

pytorch and chainer allow for dynamic graph buildingothers use static graph buildingstrengths/weaknesses: gpu, distributed computing, model building flexibility,debug...






Beyond Tensorflow

Other autodiff softwares: Pytorch/Chainer/Theano/Caffepytorch and chainer allow for dynamic graph building

others use static graph buildingstrengths/weaknesses: gpu, distributed computing, model building flexibility,debug...






Beyond Tensorflow

Other autodiff softwares: Pytorch/Chainer/Theano/Caffepytorch and chainer allow for dynamic graph buildingothers use static graph building

strengths/weaknesses: gpu, distributed computing, model building flexibility,debug...






Beyond Tensorflow







Beyond Tensorflow


High level interface: Keras

use tf and theano as backendspecify model architecture more easily





Beyond Tensorflow


High level interface: Kerasuse tf and theano as backend

specify model architecture more easily





Beyond Tensorflow







Beyond Tensorflow



High level interface: stick with Tensorflow

tensorflow.contrib




Beyond Tensorflow







Beyond Tensorflow

() low level vs. high level




Additional Resources

TF official website: https://www.tensorflow.org/

TF Basic tutorial: https://www.tensorflow.org/tutorials/

Keras doc: https://keras.io/

TF tutorials: https://github.com/aymericdamien/TensorFlow-Examples




































OpenAI Gym - very brief intro

() classic control () humanoid () atari game - pong





Gym is a testbed for RL algorithms. Doc: https://github.com/openai/gym

make an environment: env = gym.make(”CartPole-v0”)

initialize: obs = env.reset()first obs, we take the first action based on this

take a step: obs, reward, done, info = env.step(action)based on obs, we make choices on which action to takereward is the reward collected during this one step transitiondone is True or False, to indicate if the episode terminatesinfo additional info about the env, normally empty dictionary

display state: env.render(). Only display current state, need to render in a loopto display consecutive states.

























initialize: obs = env.reset()

first obs, we take the first action based on this



















take a step: obs, reward, done, info = env.step(action)

based on obs, we make choices on which action to takereward is the reward collected during this one step transitiondone is True or False, to indicate if the episode terminatesinfo additional info about the env, normally empty dictionary









take a step: obs, reward, done, info = env.step(action)based on obs, we make choices on which action to take

reward is the reward collected during this one step transitiondone is True or False, to indicate if the episode terminatesinfo additional info about the env, normally empty dictionary









take a step: obs, reward, done, info = env.step(action)based on obs, we make choices on which action to takereward is the reward collected during this one step transition

done is True or False, to indicate if the episode terminatesinfo additional info about the env, normally empty dictionary









take a step: obs, reward, done, info = env.step(action)based on obs, we make choices on which action to takereward is the reward collected during this one step transitiondone is True or False, to indicate if the episode terminates

info additional info about the env, normally empty dictionary























Beyond OpenAI Gym

There are many other open source projects that provide RL benchmarks

Continuous control: DeepMind Control Suite, Roboschool

Gym-like: rllab

Video game: VizDoom

Cooler video game: DeepMind StarCraft II Learning Environment

Or build your own environments...




Beyond OpenAI Gym



Gym-like: rllab

Video game: VizDoom






Beyond OpenAI Gym



Gym-like: rllab

Video game: VizDoom






Beyond OpenAI Gym



Gym-like: rllab

Video game: VizDoom






Beyond OpenAI Gym



Gym-like: rllab

Video game: VizDoom






Basic DQN [Mnih, 2013]

MDP with state action space S ,A

Instant reward r

Policy π : S 7→ A

Maximize cumulative reward Eπ[∑∞

t=0 rtγt ]

Action value function Qπ(s, a) for state s, action a, under policy π.

Bellman errorE [(Qπ(st , at)−max

aE [rt + γQπ(st+1, a))2]

Bellman error is zero iff optimal policy Q∗(s, a)






Instant reward r



t=0 rtγt ]










Instant reward r



t=0 rtγt ]










Instant reward r



t=0 rtγt ]










Instant reward r



t=0 rtγt ]










Instant reward r



t=0 rtγt ]










Instant reward r



t=0 rtγt ]










Instant reward r



t=0 rtγt ]








Basic DQN: Conventional Q learning

Start with any Q vector Q(0)

Contractive operator T defined as

TQ(st , at) =def maxa

E [rt + γQ(st+1, a)]

Apply contractive operatorQ(t+1) ← TQ(t)

WIll converge under mild conditions. Converge to the fixed point

Q = TQ

which gives zero bellman error.











Q = TQ












Q = TQ












Q = TQ












Q = TQ





Basic DQN

Approximate using neural net Qθ(s, a) ≈ Qπ(s, a) with parameter θ.

Discrete action space |A| <∞: input state s, output |A| values, the ith valuerepresents the approximate action-value function for the ith action.

Continuous action space |A| =∞: input state s and action a, output one valuewhich represents Qπ(s, a). Not our focus here.

Minimize Bellman error

Eπ[(Qθ(si , ai )− ri −maxa

Qθ(s ′i , a))2]

Sample based, given tuples {(si , ai , ri , s ′i )}Ni=1

minθ

N∑i=1

1

N(Qθ(si , ai )− ri −max

aQθ(s ′i , a))2




Basic DQN






Qθ(s ′i , a))2]


minθ

N∑i=1

1


aQθ(s ′i , a))2




Basic DQN






Qθ(s ′i , a))2]


minθ

N∑i=1

1


aQθ(s ′i , a))2




Basic DQN






Qθ(s ′i , a))2]


minθ

N∑i=1

1


aQθ(s ′i , a))2




Basic DQN






Qθ(s ′i , a))2]


minθ

N∑i=1

1


aQθ(s ′i , a))2




Basic DQN






Qθ(s ′i , a))2]


minθ

N∑i=1

1


aQθ(s ′i , a))2




Basic DQN

Instablity in optimizing

minθ

N∑i=1

1


aQθ(s ′i , a))2

Two techniques to alleviate instability: experience replay and target network.

Experience Replay: store experience tuple {si , ai , ri , s ′i } into replay buffer B,when training, sample batches of experience {si , ai , ri , s ′i }bi=1 from B and updateparameters using SGD

Target Network: the training target ri + maxa Qθ(s ′i , a) is non-stationary. Makeit stationary by introducing a slowly updated target net θ− and compute target asri + maxa Qθ−(s ′i , a)




Basic DQN


minθ

N∑i=1

1


aQθ(s ′i , a))2







Basic DQN


minθ

N∑i=1

1


aQθ(s ′i , a))2







Basic DQN


minθ

N∑i=1

1


aQθ(s ′i , a))2







Basic DQN


minθ

N∑i=1

1


aQθ(s ′i , a))2







Basic DQN

Naive exploration ε−greedyin state s, with prob ε take action randomly; with prob 1− ε take greedy action

arg maxa

Qθ(s, a)

More advanced exploration: noisy net, parameter space noise, bayesian updates...

Learning rate: constant α = .001 for example, not RM scheme.




Basic DQN

Naive exploration ε−greedy

in state s, with prob ε take action randomly; with prob 1− ε take greedy action

arg maxa

Qθ(s, a)






Basic DQN


arg maxa

Qθ(s, a)






Basic DQN


arg maxa

Qθ(s, a)






Basic DQN


arg maxa

Qθ(s, a)






Basic DQN

() dqn pseudocode




End

Thanks!


Documents

Introduction to Deep Learning with Tensor owIntroduction to Deep Learning with Tensor ow Yunhao (Robin) Tang Department of IEOR, Columbia University February 25, 2019 IEOR 8100: Reinforcement