Stochastic Dynamic Programming with Factored Representations Presentation by Dafna Shahaf...

Stochastic Dynamic Programming with Factored Representations

Presentation by Dafna Shahaf(Boutilier, Dearden, Goldszmidt 2000)

The Problem Standard MDP algorithms require explicit

state space enumeration Curse of dimensionality Need: Compact Representation

(intuition: STRIPS) Need: versions of standard dynamic

programming algorithms for it

A Glimpse of the Future

Policy Tree Value Tree

A Glimpse of the Future: Some Experimental Results

Roadmap

MDPs- Reminder Structured Representation for MDPs:

Bayesian Nets, Decision Trees Algorithms for Structured Representation Experimental Results Extensions

MDPs- Reminder

(states, actions, transitions, rewards)

Discounted infinite-horizon Stationary Policies

(an action to take at state s) Value functions: is k-stage-to-go

value function for π)(sV k

AS :

RTAS ,,,

Roadmap

MDPs- Reminder Structured Representation for MDPs:

Bayesian Nets, Decision Trees Algorithms for Structured Representation Experimental Results Extensions

Representing MDPs as Bayesian Networks: Coffee world

O: Robot is in office W: Robot is wet U: Has umbrella R: It is raining HCR: Robot has coffee HCO: Owner has coffee

Go: Switch location BuyC: Buy coffee DelC: Deliver coffee GetU: Get umbrella

The effect of the actions might be noisy.Need to provide a distribution for each effect.

Representing Actions: DelC

00.300

Representing Actions: Interesting Points

No need to provide marginal distribution over pre-action variables

Markov Property: we need only the previous state For now, no synchronic arcs Frame Problem? Single Network vs. a network for each action Why Decision Trees?

Representing Reward

Generally determined by a subset of features.

Policies and Value Functions

Policy Tree Value Tree

The optimal choice may depend only on certain variables (given some others).

FeaturesHCR=T

HCR=F

ValuesActions

Roadmap

MDPs- Reminder Structured Representation for MDPs:

Bayesian Nets, Decision Trees Algorithms for Structured Representation Experimental Results Extensions

Bellman Backup

Q-Function: The value of performing a in s, given value function v

Value Iteration- Reminder

)'(' )',,'Pr()()( ss vsassRsQva

)(max:)(max)(1

sQsQsV kaa

Vaa

kk

)}'(' ),,'Pr({max)()( 1 ss VsassRsV ka

k

Structured Value Iteration- OverviewInput: Tree( ). Output: Tree( ).

1. Set Tree( )= Tree( )

2. Repeat

(a) Compute Tree( )= Regress(Tree( ),a)

for each action a

(b) Merge (via maximization) trees Tree( )

to obtain Tree( )

Until termination criterion. Return Tree( ).

VkaQ

VkaQ

RR

0V

1kV1kV

kV

*V

Example World

Step 2a: Calculating Q-Functions

)'(' )',,Pr()()( ss VsassRsQVa

1. Expected FutureValue

2. DiscountingFutureValue

3. AddingImmediate

Reward

How to use the structure of the trees?

Tree( ) should distinguish only conditions under which a makes a branch of Tree(V) true with different odds.

VaQ

Calculating :

Tree(V0)

1aQ

PTree( )

Finding conditions under which a will have distinct expected value, with respect to V0

1aQ FVTree( )1aQ

Undiscounted Expected Future Value for performing action a with one-stage-to-go.

Tree( )1aQ

Discounting FVTree (by 0.9), and adding the immediate reward function.

Z:

Z: Z:

1*10+0*0

An Alternative View:

(a more complicated example)

Tree(V1) PartialPTree( )

UnsimplifiedPTree( )

PTree( )

2aQ

2aQ

2aQ FVTree( )2aQ Tree( )

2aQ

The Algorithm: Regress

Input: Tree(V), action a. Output: Tree( )

1. PTree( )= PRegress(Tree(V),a) (simplified)

VaQ

VaQ

The Algorithm: Regress

Input: Tree(V), action a. Output: Tree( )

1. PTree( )= PRegress(Tree(V),a) (simplified)

2. Construct FVTree( ):

for each branch b of PTree, with leaf node l(b)

(a) Prb =the product of individual distr. from l(b)

(b)

(c) Re-label leaf l(b) with vb.

VaQ

VaQ

VaQ

)(')'()'(Pr

VTreeb

bb bVbv

The Algorithm: Regress

Input: Tree(V), action a. Output: Tree( )

1. PTree( )= PRegress(Tree(V),a) (simplified)

2. Construct FVTree( ):

for each branch b of PTree, with leaf node l(b)

(a) Prb =the product of individual distr. from l(b)

(b)

(c) Re-label leaf l(b) with vb.

3. Discount FVTree( ) with , append Tree(R)

4. Return FVTree( )

VaQ

VaQ

VaQ

)(')'()'(Pr

VTreeb

bb bVbv

VaQ

VaQ

The Algorithm: PRegressInput: Tree(V), action a. Output: PTree( )

1. If Tree(V) is a single node, return emptyTree

2. X = the variable at the root of Tree(V)

= the tree for CPT(X) (label leaves with X)

VaQ

PXT

The Algorithm: PRegressInput: Tree(V), action a. Output: PTree( )

1. If Tree(V) is a single node, return emptyTree

2. X = the variable at the root of Tree(V)

= the tree for CPT(X) (label leaves with X)

3. = the subtrees of Tree(V) for X=t, X=f

4. = call PRegress on

VaQ

PXT

VfX

VtX TT ,

PfX

PtX TT ,

VfX

VtX TT ,

The Algorithm: PRegressInput: Tree(V), action a. Output: PTree( )

1. If Tree(V) is a single node, return emptyTree

2. X = the variable at the root of Tree(V)

= the tree for CPT(X) (label leaves with X)

3. = the subtrees of Tree(V) for X=t, X=f

4. = call PRegress on

5. For each leaf l in , add or both (according to distribution. Use union to combine labels)

6. Return

VaQ

PXT

VfX

VtX TT ,

PfX

PtX TT ,

VfX

VtX TT ,

PXT

PfX

PtX TT ,

PXT

Step 2b. Maximization

Value Iteration Complete.

Roadmap

MDPs- Reminder Structured Representation for MDPs:

Bayesian Nets, Decision Trees Algorithms for Structured Representation Experimental Results Extensions

Experimental Results

WorstCase:

BestCase:

Roadmap

MDPs- Reminder Structured Representation for MDPs:

Bayesian Nets, Decision Trees Algorithms for Structured Representation Experimental Results Extensions

Extensions

Synchronic edges POMDPs Rewards Approximation

Questions?

Backup slides

Here be dragons.

Regression through a Policy

Improving Policies: Example

Maximization Step, Improved Policy