Collaborative Reinforcement Learning

Presented by Dr. Ying Lu

Credits

Reinforcement Learning: A User’s Guide. Bill Smart at ICAC 2005

Jim Dowling, Eoin Curran, Raymond Cunningham and Vinny Cahill, "Collaborative Reinforcement Learning of Autonomic Behaviour", 2nd International Workshop on Self-Adaptive and Autonomic Computing Systems, pages 700-704, 2004. [Winner Best Paper Award].

What is RL?

“a way of programming agents by reward and punishment without needing to specify how the

task is to be achieved”

[Kaelbling, Littman, & Moore, 96]

Basic RL Model

1. Observe state, st

2. Decide on an action, at

3. Perform action

4. Observe new state, st+1

5. Observe reward, rt+1

6. Learn from experience7. Repeat

Goal: Find a control policy that will maximize the observed rewards over the lifetime of the agent

AS R

World

An Example: Gridworld

Canonical RL domain• States are grid cells• 4 actions: N, S, E, W• Reward for entering top right cell• -0.01 for every other move

Maximizing sum of rewards Shortest path• In this instance

+1

The Promise of RL

Specify what to do, but not how to do it• Through the reward function• Learning “fills in the details”

Better final solutions• Based on actual experiences, not programmer

assumptions

Less (human) time needed for a good solution

Mathematics of RL

Before we talk about RL, we need to cover some background material

• Some simple decision theory• Markov Decision Processes• Value functions

Making Single Decisions

Single decision to be made• Multiple discrete actions• Each action has a reward associated

with it

Goal is to maximize reward• Not hard: just pick the action with the largest reward

State 0 has a value of 2• Sum of rewards from taking the best action from the

state

0

1

2

A

B2

1

Markov Decision Processes

We can generalize the previous example to multiple sequential decisions

• Each decision affects subsequent decisions

This is formally modeled by a Markov Decision Process (MDP)

0

1

2

A

B2

1

5

3

4

AA

-1000

1

A A10

1B

1

Markov Decision Processes

Formally, an MDP is• A set of states, S = {s1, s2, ... , sn}

• A set of actions, A = {a1, a2, ... , am}

• A reward function, R: SAS→• A transition function,

We want to learn a policy, : S →A• Maximize sum of rewards we see over our lifetime

aai,s|jsPP tt1taij

Policies

There are 3 policies for this MDP1. 0 →1 →3 →5

2. 0 →1 →4 →5

3. 0 →2 →4 →5

Which is the best one?

0

1

2

A

B2

1

5

3

4

AA

-1000

1

A A10

1B

1

Comparing Policies

Order policies by how much reward they see1. 0 →1 →3 →5 = 1 + 1 + 1 = 3

2. 0 →1 →4 →5 = 1 + 1 + 10 = 12

3. 0 →2 →4 →5 = 2 – 1000 + 10 = -988

0

1

2

A

B2

1

5

3

4

AA

-1000

1

A A10

1B

1

Value Functions

We can define value without specifying the policy• Specify the value of taking action a from state s and

then performing optimally• This is the state-action value function, Q

0

1

2

A

B2

1

5

3

4

A

-1000

1

A10

1B

1

Q(0, A) = 12 Q(0, B) = -988

Q(3, A) = 1

Q(4, A) = 10

Q(1, A) = 2Q(1, B) = 11

Q(2, A) = -990

A

A

How do you tell whichaction to take from

each state?

Value Functions

So, we have value function• Q(s, a) = R(s, a, s’) + maxa’ Q(s’, a’)

In the form of• Next reward plus the best I can do from the next state

These extend to probabilistic actions•

s’ is thenext state

a' ,s'Q maxs' a, s,RPas,Q a's'

as's,

Getting the Policy

If we have the value function, then finding the best policy is easy

• (s) = arg maxa Q(s, a)

We’re looking for the optimal policy, (s)• No policy generates more reward than

Optimal policy defines optimal value functions•

The easiest way to learn the optimal policy is to learn the optimal value function first

a' ,s'Qmaxs' a, s,Ras,Q *a'*

Collaborative Reinforcement Learningto Adaptively Optimize MANET Routing

Jim Dowling, Eoin Curran, Raymond Cunningham and Vinny Cahill

Overview

Building autonomic distributed systems with self* properties

• Self-Organizing• Self-Healing• Self-Optimizing

Add collaborative learning mechanism to self-adaptive component modelImproved ad-hoc routing protocol

Introduction

Autonomous distributed systems will consist of interacting components free from human interference

• Existing top-down management and programming solutions require too much global state

• Bottom up, decentralized collection of components who make their own decisions based on local information

• System wide self* behavior emerges from interactions

Self-* Behavior

Self-adaptive components that change structure and/or behavior at run-time, adapt to

• discovered faults• reduced performance

Requires active monitoring of component states and external dependencies

Self-* Distributed Systems using Distributed (collaborative) Reinforcement Learning

For complex systems, programmers cannot be expected to describe all conditions

• Self-adaptive behavior learnt by components• Decentralized co-ordination of components to

support system-wide properties• Distributed Reinforcement Learning (DRL) is

extension to RL and uses neighbor interactions only

Model-Based Reinforcement Learning

( , ) ( , ) ' | , . 's

Q s a R s a P s s a V s

S

MDPAdaptationContract

AMM

action (at)rt+1

st+1

reward (rt)

state (st)

Component

1.Action Reward 2. State Transition Model

3. Next State Reward

Markov Decision Process = {States }, {Actions}, R(States,Actions), (States, Actions, States)

Decentralised System Optimisation

Coordinating the solution to a set of Discrete Optimisation Problems (DOPs)

• Components have a Partial System View• Coordination Actions

• Actions ={delegation} U {DOP actions} U {discovery}

• Connection Costs

A

B

Causally-Connected

States

CDelegation

Collaborative Reinforcement Learning

Advertisement• Update Partial Views of Neighbours

Decay• Negative Feedback on State Values in the Absence of

Advertisements

( , ) ( , ) ' | , . ' ' | ,i i i is

Q s a R s a P s s a Decay V s D s s a

S

Action Reward State Transition Model

CachedNeighbour’s V-value

ConnectionCost

Adaptation in CRL System

A feedback process to

• Changes in the optimal policy of any RL agent• Changes in the system environment• The passing time

SAMPLE: Ad-hoc Routing using DRL

Probabilistic ad-hoc routing protocol based on DRL• Adaptation of network traffic around areas of congestion

• Exploitation of stable routes

Routing decisions based on local information and information obtained from neighbors

Outperforms Ad-hoc On Demand Distance Vector Routing (AODV) and Dynamic Source Routing (DSR)

SAMPLE: A CRL System (I)

SAMPLE: A CRL System (II)

Instead of always choosing the neighbor with the best Q value, i.e., taking the delegation action

a= arg maxaQi(B, a),

a neighbor is chosen probabilistically

SAMPLE: A CRL System (III)

Pi(s’|s, aj) = E(CS/CA)

SAMPLE: A CRL System (IV)

Performance

Metric:• Maximize

• throughput

• ratio of delivered packets to undelivered packets

• Minimize• number of transmission required per packet sent

Figures 5-10

Questions/Discussions