Model-based Bayesian Reinforcement Learningin Partially Observable Domains

by

Pascal Poupart and Nikos Vlassis

(2008 International Symposium on Artificial Intelligence and Math)

Presented by Lihan He

ECE, Duke University

Oct 3, 2008

Introduction

POMDP represented as dynamic decision network (DDN)

Partially observable reinforcement learning

Belief update

Value function and optimal action

Partially observable BEETLE

Offline policy optimization

Online policy execution

Conclusion

Outline

1/14

Introduction

Final objective: learn optimal actions (policy) to achieve best reward

POMDP: partially observable Markov decision process represented by sequential decision-making problem

ROTAS ,,,,,

Reinforcement learning for POMDP: solve the decision-making problem given feedback from environment, when the dynamics of the environment (T and O) are unknown.

given action-observation sequence as history model-based: explicitly model the environment model-free: avoid to explicitly model the environment online learning: policy learning and execution at the same time offline learning: learn policy first given training data, and then

execute policy without modifying the policy

2/14

Introduction

This paper:

Bayesian model-based approach Set the prior for belief as mixture of products of Dirichlets The posterior belief is a mixture of products of Dirichlets The value function is also a mixture of products of Dirichlets The number of the mixture components increases exponentially with

respect to the time step PO-BEETLE algorithm

3/14

POMDP and DDN

Redefine POMDP as dynamic decision network (DDN)

EXXG ,',

},,,,{ RDCBAX

},,,{ RDCBS }{DO SO }{RR SR

X, X’ : two consecutive time steps

Observation and reward are subsets of state variableThe conditional probability distributions of state Pr(s’|pas’) jointly

encode the transition, observation and reward models T, O and R

4/14

POMDP and DDN

The optimal value function satisfies Bellman’s equation

Given X, S, R, O, A, edge E and the dynamics Pr(s’|pas’):

Belief update:

Objective: finding a policy that maximizes the expected total reward

Value iteration algorithms optimize the value function by iteratively computing the right hand side of the Bellman’s equation.

5/14

POMDP and DDN

For reinforcement learning, assume X, S, R, O, A are known, and edges E are known, but the dynamics Pr(s’|pas’) are unknown.

We augment graph: Dynamics are included in the graph, denoted by parameter Θ.

If the unknown model is static,

Belief over s joint belief over s and θ

6/14

PORL: belief update

Problem: number of mixture components increases by a factor of |S| (exponential growth with time)

Prior setting for belief: a mixture of products of Dirichlets

Posterior belief (after taking action a and receiving observation o’) is again a mixture of products of Dirichlets

7/14

PORL: value function and optimal action

The augmented POMDP is hybrid, with discrete state variables S and continuous model variables Θ

Discrete state POMDP: )(max)(* bbV

s

sbsab )()()(with

Continuous state POMDP [1]: dssbsabs )()()(

[1] Porta, J. M.; Vlassis, N. A.; Spaaan, M. T. J.; and Poupart, P. 2006. Point-based value iteration for continuous POMDPs. Journal of Machine Learning Research 7:2329–2367.

The α-function α(s,θ) can also be represented as a mixture of products of Dirichlets

Hybrid state POMDP: s

dsbsab

),(),()(

8/14

PORL: value function and optimal action

Assume for k step-to-go is )(bV k

then for k+1 step-to-go is )(1 bV k

decomposed in 3 steps

find optimal action for belief b

find the corresponding α-function

with

Problem: number of mixture components increases by a factor of |S| (exponential growth with time) 9/14

1)

2)

3)

PO-BEETLE: offline policy optimization

Policy learning is performed offline, given sufficient training data (action-observation sequence)

10/14

PO-BEETLE: offline policy optimization

Keep the number of mixture components for α-functions bounded:

Approach 1: approximation using basis functions

Approach 2: approximation by important components

11/14

PO-BEETLE: online policy execution

Given policy, the agent executes the policy and updates belief online.

Keep the number of mixture components for belief b bounded:

Approach 1: approximation using importance sampling

12/14

PO-BEETLE: online policy execution

Approach 2: particle filtering: simultaneously update belief and reduce the number of mixture components

Sample one updated component (after taking a and receiving o’)

The updated belief is represented by k particles

13/14

Conclusion

Bayesian model-based reinforcement learning; Prior belief is a mixture of products of Dirichlets; Posterior belief is also a mixture of products of Dirichlets,

with the number of mixture components growing exponentially with time;

α-functions (associated with value functions) are also represented as mixtures of products of Dirichlets that grow exponentially with time;

Partially observable BEETLE algorithm.

14/14