Unsupervised Methods For Subgoal Discovery During Intrinsic...

Unsupervised Methods For Subgoal Discovery During

Intrinsic Motivation in Model-Free Hierarchical Reinforcement Learning

Jacob Rafati http://rafati.net

Ph.D. Candidate Electrical Engineering and Computer Science

Computational Cognitive Neuroscience Laboratory University of California, Merced

Co-authored with: David C. Noelle

Goals & Rules• “Key components of games are goals, rules, challenge,

and interaction. Games generally involve mental or physical stimulation, and often both.”

https://en.wikipedia.org/wiki/Game

Reinforcement LearningReinforcement learning (RL) is learning how to map situations (state) to actions so as to maximize numerical reward signals received during the experiences that an artificial agent has as it interacts with its environment.

et = {st, at, st+1, rt+1}experience:

(Sutton and Barto, 2017)

Objective: Learn ⇡ : S ! A to maximize cumulative rewards

Super-Human Success

(Mnih. et. al., 2015)

Failure in a complex task

Learning Representations in Hierarchical Reinforcement Learning

• Trade-off between exploration and exploitation in an environment with sparse feedback is a major challenge.

• Learning to operate over different levels of temporal abstraction is an important open problem in reinforcement learning.

• Exploring the state-space while learning reusable skills through intrinsic motivation.

• Discovering useful subgoals in large-scale hierarchical reinforcement learning is a major open problem.

ReturnReturn is the cumulative sum of a received reward:

st st+1

st�1

rtrt�1

atat�1s0 sT

Gt =TX

t0=t+1

�t0�t�1rt0

� 2 [0, 1] is the discount factor

Policy Function• Policy Function: At each time step agent implements a

mapping from states to possible actions

⇡ : S ! A

• Objective: Finding an optimal policy that maximizes the cumulated rewards

⇡⇤ = argmax⇡

E⇥Gt|St = s

⇤, 8s 2 S

Q-Function• State-Action Value Function is the expected return when

starting from (s,a) and following a policy thereafter

Q⇡ : S ⇥A ! R

Q⇡(s, a) = E⇡[Gt|St = s,At = a]

Temporal Difference• Model-free reinforcement learning algorithm.

• State-transition probabilities or reward function are not available

• A powerful computational cognitive neuroscience model of learning in brain

• A combination of Monte Carlo method and Dynamic Programming

Q-learning Q(s, a) Q(s, a) + ↵[r + �max

a0Q(s0, a0)�Q(s, a)]

Q(s, a) ! prediction of return

r + �maxa0

Q(s0, a0) ! target value

State Function Approximator

state-action Values

...q(s, ai;w)

Generalization

w q(s, a;w)s

Q(s, a) ⇡ q(s, a;w)

Deep RLminw

w = argminw

L(w) = E(s,a,r,s0)⇠D

h�r +max

a0q(s0, a0;w�)� q(s, a;w)

D = {et|t = 0, . . . , T} ! Experience replay memory

Stochastic Gradient Decent method

w w �rwL(w)

Q-Learning with experience replay memory

Failure: Sparse feedback

(Botvinick et al., 2009) Subgoals

Complex Task

Simple Tasks

Major Goals Minor Goals Actions

Hierarchy in Human Behavior & Brain Structure

Hierarchical Reinforcement Learning Subproblems

• Subproblem 1: Learning a meta-policy to choose a subgoal

• Subproblem 2: Developing skills through intrinsic motivation

• Subproblem 3: Subgoal discovery

Environment Meta-controller

Controller

st+1, rt+1

Critic

r̃t+1atat

Meta-controller/Controller Framework

Kulkarni et. al. 2016

Subproblem 1: Temporal Abstraction

Room 1Room 2

Room 3 Room 4

Rooms Task

Subproblem 2. Developing skills through Intrinsic Motivation

……………

…… Gaussianrepresentation

fully connected

distributed

conjunctive

representation

q(st, gt, a;w)

State-Goal Q Function

Room 1Room 2

Room 3 Room 4

Room-1

Room-3

Room-4

Room-2

Reusing the skills

0 0.2 0.4 0.6 0.8 1x

Grid World Task with Key and Door

Reusing the skillsRoom-1

Room-3

Room-4

Room-2

0 0.2 0.4 0.6 0.8 1x

Room-1

Room-3

Room-4

Room-2

Reusing the skills

0 0.2 0.4 0.6 0.8 1x

Reusing the skillsRoom-1

Room-3

Room-4

Room-2

0 0.2 0.4 0.6 0.8 1x

Room-1

Room-3

Room-4

Room-2

Reusing the skills

0 0.2 0.4 0.6 0.8 1x

Room-1

Room-3

Room-4

Room-2

Reusing the skills

Subproblem 3. Subgoal Discovery

(Sismek et al., 2005)

(Goel and Huber, 2003)

(Machado, et. al. 2017)

finding proper G

• Purpose: Discovering promising states to pursue, i.e. finding

• Implementing subgoal discovery algorithm for large-scale model free reinforcement learning problem

• No access to MDP models (state-transition probabilities, environment reward function, State space)

Subproblem 3. Candidate Subgoals

• It is close (in terms of actions) to a rewarding state.

• It represents a set of states, at least some of which tend to be along a state transition path to a rewarding state.

• Unsupervised learning (clustering) on the limited past experience memory collected during intrinsic motivation

• Centroids of clusters are useful subgoals (e.g. rooms)

• Detecting outliers as potential subgoals (e.g. key, box)

• Boundary of two clusters can lead to subgoals (e.g. doorway between rooms)

Unsupervised Subgoal Discovery

Unification of Hierarchical Reinforcement Learning Subproblems

• Implementing a hierarchical reinforcement learning framework that makes it possible to simultaneously perform subgoal discovery, learn appropriate intrinsic motivation, and succeed at meta-policy learning

• The unification element is using experience replay memory D

Model-Free HRL

0 20000 40000 60000 80000 100000

Training steps

Successin

ReachingSu

bgoals%

0 20000 40000 60000 80000 100000

Training steps

Successin

SolvingTask%

Our Unified Model-Free HRL Method

Regular RL

0 20000 40000 60000 80000 100000

Training steps

EpisodeReturn

Regular RL

Montezuma’s RevengeMeta-Controller Controller

Montezuma’s Revenge

0 500000 1000000 1500000 2000000 2500000

Training steps

Successin

reachingsubgoals%

DeepMind DQN Algorithm (Mnih et. al., 2015)

0 500000 1000000 1500000 2000000 2500000

Training steps

Average

return

10episdes

DeepMind DQN Algorithm (Mnih et. al., 2015)

Conclusions• Unsupervised Learning can be used to discover

useful subgoals in games.

• Subgoals can be discovered using model-free methods.

• Learning in multiple levels of temporal abstraction is the key to solve games with sparse delayed feedback.

• Intrinsic motivation learning and subgoal discovery can be unified in model-free HRL framework.

References• Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A.,

Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015). Human-level control through deep reinforcement learning. Nature, 518(7540):529–533.

• Sutton, R. S., and Barto, A. G. (2017). Reinforcement Learning: An Introduction. MIT Press. 2nd edition.

• Botvinick, M. M., Niv, Y., and Barto, A. C. (2009). Hierarchically organized behavior and its neural foundations: A reinforcement learning perspective. Cognition, 113(3):262 – 280.

• Goel, S. and Huber, M. (2003). Subgoal discovery for hierarchical reinforcement learning using learned policies. In Russell, I. and Haller, S. M., editors, FLAIRS Conference, pages 346–350. AAAI Press.

• Kulkarni, T. D., Narasimhan, K., Saeedi, A., and Tenenbaum, J. B. (2016). Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. NeurIPS 2016.

• Machado, M. C., Bellemare, M. G., and Bowling, M. H. (2017). A laplacian framework for option discovery in reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, pages 2295–2304.

• Sutton, R. S., Precup, D., and Singh, S. (1999). Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112(1):181 – 211.

Slides, Paper, and Code:

http://rafati.net

Poster Session on Wednesday.

Unsupervised Methods For Subgoal Discovery During Intrinsic...

Documents

From Reinforcement Learning to Deep Reinforcement …fagostin/assets/files/...Keywords: Machine learning · Reinforcement learning Deep learning · Deep reinforcement learning 1 Introduction

Hierarchical Reinforcement Learning with Unlimited Recursive ...Hierarchical reinforcement learning architecture RGoal • In RGoal, an agent's subgoal settings are similar to subroutine

Deep Learning for Reinforcement Learning in Pacman · Deep Learning for Reinforcement Learning in Pacman Deep Learning für Reinforcement Learning in Pacman Vorgelegte Bachelor-Thesis

Inverse Reinforcement Learning CS885 Reinforcement

Reinforcement Learning Introduction Passive Reinforcement Learning Temporal Difference Learning Active Reinforcement Learning Applications Summary

Hierarchical Deep Reinforcement Learning: Integrating ...papers.nips.cc/paper/6233-hierarchical-deep-reinforcement-learning... · work that integrates deep reinforcement learning

Tutorial: Deep Reinforcement Learning - Machine Learning ...hunch.net/~beygel/deep_rl_tutorial.pdfTutorial: Deep Reinforcement Learning - Machine Learning

Multi-Objective Reinforcement Learning using Sets of Pareto … · 2020. 10. 19. · learning and multi-objective reinforcement learning. 2.1 Reinforcement Learning A reinforcement

Subgoal 3A: Utilize HUD assistance to improve educational outcomes and early learning and development. Subgoal 3B: Utilize HUD assistance to improve

Inverse Reinforcement Learning - Peoplecbfinn/_files/bootcamp_inverserl.pdf · Apprenticeship Learning via Inverse Reinforcement Learning. Good introduction to inverse reinforcement

Reinforcement Learning Chapter 13 What is Reinforcement Learning? Q-Learning Examples 1

Eick: Reinforcement Learning. Reinforcement Learning Introduction Passive Reinforcement Learning Temporal Difference Learning Active Reinforcement Learning

Universal Reinforcement Learning Algorithms: … Reinforcement Learning Algorithms: Survey and ... class of models hypothesesenvironments M . ... Universal Reinforcement Learning Algorithms:

Cooperative Inverse Reinforcement Learning...Cooperative Inverse Reinforcement Learning Dylan Hadfield-Menell CS237: Reinforcement Learning May 31, 2017

Reinforcement Learning & Apprenticeship Learning

Reinforcement Learning and Deep Reinforcement Learningcse.ucdenver.edu/.../Class-22-Reinforcement-learning-DL.pdf · 2018. 11. 28. · Outlines 1 Principles of Reinforcement Learning

Reinforcement Learning Lecture Inverse Reinforcement Learningipvs.informatik.uni-stuttgart.de/mlr/wp-content/uploads/2017/07/09... · Reinforcement Learning Inverse Reinforcement

Reinforcement Learning - uni-freiburg.degki.informatik.uni-freiburg.de/.../recordings/reinforcement.pdf · Reinforcement Learning 3 What is Reinforcement Learning? Learning from interaction

Reinforcement Learning: Learning algorithms

Reinforcement Learning - Multi-Agent Reinforcement