Markov Tutorial CDC Shanghai 2009

Lyapunov functions, value functions,

Sean Meyn

Department of Electrical and Computer EngineeringUniversity of Illinois and the Coordinated Science Laboratory

Joint work with R. Tweedie, I. Kontoyiannis, and P. Mehta

Supported in part by NSF (ECS 05 23620, and prior funding), and AFOSR

and performance bounds

Objectives

Nonlinear state space model ≡ (controlled) Markov process,

Typical form:

noisecontrol

dX(t) = f(X(t), U(t)) dt + σ(X(t), U(t)) dW (t)

state process X

Objectives

Nonlinear state space model ≡ (controlled) Markov process,

Typical form:

Questions: For a given feedback law,

• Is the state process stable?

• Is the average cost finite?

• Can we solve the DP equations?

• Can we approximate the average cost η ? The value function h ?

noisecontrol

E[c(X(t), U(t))]

dX(t) = f(X(t), U(t)) dt + σ(X(t), U(t)) dW (t)

minu

c(x, u) + Duh∗

∗∗

(x) = η∗{ }

state process X

Outline

Markov Models

Representations

Lyapunov Theory

Conclusionsπ(f

)<

∞

DV (x) ≤ −f(x) + bIC(x)

‖P t (x, · ) − π‖f → 0

sup

CE

x [Sτ

C(f

)]<

∞

IMarkov Models

Notation

Markov chain:

Countable state space, X

Transition semigroup,

X = X(t) : t ≥ 0{ }

P t(x, y) = P X(s + t) = y X(s) = x{ }

x, y, ∈ X

Notation: Generators & Resolvents

Markov chain:

Countable state space, X

Transition semigroup,

Generator: For some domain of functions h,

X = X(t) : t ≥ 0{ }

Dh (x) = limt→0

1

tE[h(X(s + t)) − h(X(s)) X(s) = x]

= limt→0

1

t(P th (x) − h(x))

P t(x, y) = P X(s + t) = y X(s) = x{ }

x, y, ∈ X

Notation: Generators & Resolvents

Generator: For some domain of functions h,

Rate matrix:

P t = eQt

Dh (x) = limt→0

1

tE[h(X(s + t)) − h(X(s)) X(s) = x]

= limt→0

1

t(P th (x) − h(x))

Dh (x) =∑

y

Q(x, y)h(y)

µα

Example: MM1 Queue

Sample paths:

Rate matrix:

X(t + ε) ≈

x + 1 Prob εα

x − 1 Prob εµ

x Prob 1 − ε(α + µ)

Q =

−α α 0 0 0 0 · · ·µ −α − µ α 0 0 0 · · ·0 µ −α − µ α 0 0 · · ·0 0 µ −α − µ α 0 · · ·0 0 0 µ −α − µ α · · ·0 0 0 0 µ −α − µ · · ·...

......

......

...

σ2W = 0 σ2

W = 1

Example: O-U Model

Sample paths:

Generator:

dX(t) = AX(t) dt + B dW (t)

A n × n, Bn× 1, W standard BM

Dh (x) = (Ax)T∇h (x) + BT∇2h (x)B

σ2W = 0 σ2

W = 1

Example: O-U Model

Sample paths:

Generator:

h quadratic,

dX(t) = AX(t) dt + B dW (t)

A n × n, Bn× 1, W standard BM

Dh (x) = (Ax)T∇h (x) + BT∇2h (x)B

h(x) = 12xTPx ∇h (x) = Px

∇2h (x) = P

Dh (x) = 12xT(PA + ATP )x + BTPB

Notation: Generators & Resolvents

Generator: For some domain of functions h,

Rate matrix:

Resolvent:

P t = eQt

Dh (x) = limt→0

1

tE[h(X(s + t)) − h(X(s)) X(s) = x]

= limt→0

1

t(P th (x) − h(x))

Dh (x) =∑

y

Q(x, y)h(y)

Rα =

∫ ∞

0

e−αtP t

Notation: Generators & Resolvents

Generator: For some domain of functions h,

Rate matrix:

Resolvent: Resolvent equations:

P t = eQt

Dh (x) = limt→0

1

tE[h(X(s + t)) − h(X(s)) X(s) = x]

= limt→0

1

t(P th (x) − h(x))

Dh (x) =∑

y

Q(x, y)h(y)

Rα = Rα =

∫ ∞

0

e−αtP t [ Iα − Q]−1

QRα = RαQ = αRα − I

Notation: Generators & Resolvents

Motivation: Dynamic programming. For a cost function c,

h

Discounted-cost value function

α(x) = Rαc (x) =∑

y∈X

Rα(x, y)c(y)

=

∫ ∞

0

eαtE[c(X(t)) X(0) = x] dt

Notation: Generators & Resolvents

Motivation: Dynamic programming. For a cost function c,

Resolvent equation = dynamic programming equation,

h

Discounted-cost value function

α(x) = Rαc (x) =∑

y∈X

Rα(x, y)c(y)

c + Dh αα = hα

=

∫ ∞

0

eαtE[c(X(t)) X(0) = x] dt

Notation: Steady State Distribution

Invariant (probability) measure π: X is stationary. In particular,

X(t) ∼ π, t ≥ 0

Notation: Steady State Distribution

Invariant (probability) measure π: X is stationary. In particular,

Characterizations:

X(t) ∼ π, t ≥ 0

∑

x∈X

π(x)P t(x, y) = π(y)

y ∈ X

∑

x∈X

π(x)Q(x, y) = 0

α∑

x∈X

π(x)Rα(x, y) = π(y), α > 0

Notation: Relative Value Function

Invariant measure π, cost function c , steady-state mean η

Relative value function:

h(x) =

∫ ∞

0

E[c(X(t)) − η X(0) = x] dt

Notation: Relative Value Function

Invariant measure π, cost function c , steady-state mean η

Relative value function:

Solution to Poisson’s equation (average-cost DP equation):

h(x) =

∫ ∞

0

E[c(X(t)) − η X(0) = x] dt

c + Dh = η

IIRepresentations

π ∝ ν[I − (R − s ⊗ ν)]−1

h = [I − (R − s ⊗ ν)]−1c̃

Irreducibility

ψ-Irreducibility:

ψ(y) > 0 =⇒ R(x, y) > 0 all x

> 0ψ(y) > 0 =⇒ P X(t) reaches all xy X(0) = x{ }

Small Functions and Small Measures

ψ-Irreducibility:

Small functions and measures: For a function s and probability ν,

ψ(y) > 0 =⇒ R(x, y) > 0 all x

> 0ψ(y) > 0 =⇒ P X(t) reaches all xy X(0) = x{ }

R(x, y) ≥ s(x)ν(y), x, y ∈ XR =

∫ ∞

0

e−tP t dt

Small Functions and Small Measures

ψ-Irreducibility:

Small functions and measures: For a function s and probability ν,

Resolvent dominates rank-one matrix,

ψ(y) > 0 =⇒ R(x, y) > 0 all x

> 0ψ(y) > 0 =⇒ P X(t) reaches all xy X(0) = x{ }

R(x, y) ≥ s(x)ν(y), x, y ∈ X

R ≥ s ⊗ νR =

∫ ∞

0

e−tP t dt

Small Functions and Small Measures

ψ-Irreducibility:

Small functions and measures: For a function s and probability ν,

Resolvent dominates rank-one matrix,

and WLOG,

ψ-Irreducibility justi�es assumption: for all x

ψ(y) > 0 =⇒ R(x, y) > 0 all x

> 0ψ(y) > 0 =⇒ P X(t) reaches all xy X(0) = x{ }

R(x, y) ≥ s(x)ν(y), x, y ∈ X

R ≥ s ⊗ ν

s(x) > 0

ν = δx∗ , where ψ(x∗) > 0

R =

∫ ∞

0

e−tP t dt

µα

Example: MM1 Queue

R(x, y) > 0 for all x and y (irreducible in usual sense)

Conclusion:

where

R(x, y ) ≥ s(x)ν(y )

s(x) := R(x, 0)

ν := δ0

σ2W = 0 σ2

W = 1

dX(t) = AX(t) dt + B dW (t)Example: O-U Model

GaussianFull rank if and only if (A, B) is controllable.

R

Conclusion: Under controllability, for any m, there is ε s.t.,

where

(0, ).

R(x, A) ≥ s(x all x and A)ν(A)

s(x) = εIν(A) uniform on

{‖x‖ ≤ m

}

{‖x‖ ≤ m

}

Potential Matrix

Potential matrix: G(x, y) =

∞∑

n=0

(R − s ⊗ ν)n (x, y)

G = [I − (R − s ⊗ ν)]−1

Representation of π

Potential matrix:

νG (y) =∑

x∈X

ν(x)G(x, y)

G(x, y) =

∞∑

n=0

(R − s ⊗ ν)n (x, y)

π ∝ νG

Representation of h

Gc̃ (y) =∑

y∈X

G(x, y)c̃(y)∑

y∈X

Potential matrix: G(x, y) =

∞∑

n=0

(R − s ⊗ ν)n (x, y)

c̃(x) = c(x) − η

η = π(x)c(x)

h = RGc̃ + constant

Representation of h

Gc̃ (y) =∑

y∈X

G(x, y)c̃(y)∑

y∈X

Potential matrix:

If sum converges, then Poisson’s equation is solved:

G(x, y) =

∞∑

n=0

(R − s ⊗ ν)n (x, y)

c̃(x) = c(x) − η

η = π(x)c(x)

c(x) + Dh (x) = η

h = RGc̃ + constant

IIILyapunov Theory

π(f

)<

∞

∆V (x) ≤ −f(x) + bIC(x)

‖P n(x, · ) − π‖f → 0

sup

CE

x [Sτ

C(f

)]<

∞

Lyapunov Functions

DV ≤ −g + bs

Lyapunov Functions

General assumptions:

DV ≤ −g + bs

V : X → (0,∞)

g : X → [1,∞)

b < ∞, s smalle.g., s(x) = IC(x), C �nite

Lyapunov Bounds on G DV ≤ −g + bs

Resolvent equation gives RV − V ≤ −Rg + bRs

Lyapunov Bounds on G DV ≤ −g + bs

Resolvent equation gives

Since s⊗ν is non-negative,

−[I − (R − s ⊗ ν)]V ≤ RV − V ≤ −Rg + bRs

RV − V ≤ −Rg + bRs

G−1

Lyapunov Bounds on G DV ≤ −g + bs

Resolvent equation gives

Since s⊗ν is non-negative,

More positivity,

Some algebra,

−[I − (R − s ⊗ ν)]V ≤ RV − V ≤ −Rg + bRs

RV − V ≤ −Rg + bRs

G−1

V ≥ GRg − bGRs

GR = G(R − s ⊗ ν) + (Gs) ⊗ ν ≥ G − I

Gs ≤ 1

Lyapunov Bounds on G DV ≤ −g + bs

Resolvent equation gives

Since s⊗ν is non-negative,

More positivity,

Some algebra,

General bound:

−[I − (R − s ⊗ ν)]V ≤ RV − V ≤ −Rg + bRs

RV − V ≤ −Rg + bRs

G−1

V ≥ GRg − bGRs

GR = G(R − s ⊗ ν) + (Gs) ⊗ ν ≥ G − I

Gs ≤ 1

GRg ≤ V + 2b

Gg ≤ V + g + 2b

Existence of π DV ≤ −g + bs

DV ≤ −1 + bsCondition (V2)

Representation:

Bound:

Conclusion: π exists as a probability measure on X

π ∝ νG

GRg ≤ V + 2b =⇒ G(x, X) ≤ V (x) + 2b

Existence of moments DV ≤ −g + bs

DV ≤ −g + bsCondition (V3)

Representation:

Bound:

π ∝ νG

Gg ≤ V + 2+ g b

Existence of moments DV ≤ −g + bs

DV ≤ −g + bsCondition (V3)

Representation:

Bound:

Conclusion: π exists as a probability measure on X

and the steady-state mean is �nite,

π ∝ νG

Gg ≤ V + 2+ g b

π(g) :=∑

x∈X

π(x)g(x) ≤ b

µα

Example: MM1 Queue ρ =α

µ

Linear Lyapunov function,

Conclusion: (V2) holds if and only if ρ < 1

DV (x) =

∞∑

y=0

Q(x, y)y

= α(x + 1) + µ(x − 1) − (α + µ)x

V (x) = x

= −(µ − α) x > 0

µα

Example: MM1 Queue ρ =α

µ

g(x) = 1 + x

QuadraticLyapunov function,

Conclusion: (V3) holds,

DV (x) =

∞∑

y=0

Q(x, y)y

V (x) = x2

2

= α(x + 1)2 + µ(x − 1)2 − (α + µ)x2

= α(x2 + 2x + 1) + µ(x2 − 2x + 1)2 − (α + µ)x2

= −2(µ − α)x + α + µ

if and only if ρ < 1

σ2W = 0 σ2

W = 1

dX(t) = AX(t) dt + B dW (t)Example: O-U Model

Suppose that P>0 solves the Lyapunov equation,

h quadratic, h(x) = 12xTPx ∇h (x) = Px

∇2h (x) = P

Dh (x) = 12xT(PA + ATP )x + BTPB

= -IPA + ATP

σ2W = 0 σ2

W = 1

dX(t) = AX(t) dt + B dW (t)Example: O-U Model

Suppose that P>0 solves the Lyapunov equation,

Then (V3) follows from the identity,

h quadratic, h(x) = 12xTPx ∇h (x) = Px

∇2h (x) = P

Dh (x) = 12xT(PA + ATP )x + BTPB

= -IPA + ATP

Dh (x) = − 12‖x‖

2 + σ2X , σ2

X = BTPB

σ2W = 0 σ2

W = 1

dX(t) = AX(t) dt + B dW (t)Example: O-U Model

Suppose that P>0 solves the Lyapunov equation,

Then (V3) follows from the identity,

The function solves Poisson’s equation,h(x) = 12xTPx

= -IPA + ATP

Dh (x) = − 12‖x‖

2 + σ2X , σ2

X = BTPB

g(x) = 12‖x‖

2

η = σ2X

Dh = −g + η

Poisson’s Equation DV ≤ −g + bs

RGg RGg≤ V + 2b =⇒ (x ) ≤ V (x) + 2b

DV ≤ −g + bsCondition (V3)

Representation:

Bound:

c̃(x) = c(x) − ηh = RGc̃ + constant

Poisson’s Equation DV ≤ −g + bs

RGg RGg≤ V + 2b =⇒ (x ) ≤ V (x) + 2b

DV ≤ −g + bsCondition (V3)

Representation:

Bound:

Conclusion: If c is bounded by g, then h is bounded,

c̃(x) = c(x) − ηh = RGc̃ + constant

h(x) ≤ V (x) + 2b

µα

Example: MM1 Queue ρ =α

µ

Poisson’s equation with

We have (V3) with V a quadratic function of x:

Recall, with

g (x) = x

Dh = −g + η

Dh (x) = −2(µ − α)x x>0+ α + µ

h (x) = x2

µα

Example: MM1 Queue ρ =α

µ

Poisson’s equation with

Solved with

g (x) = x

Dh = −g + η

h(x) = 12

x2 + x

µ − αη =

ρ

1 − ρ

IVConclusions

Final words

Just as in linear systems theory, Lyapunov functionsprovide a characterization of system properties, as wellas a practical verification tool

π(f

)<

∞

DV (x) ≤ −f(x) + bIC(x)

‖P t (x, · ) − π‖f → 0

sup

CE

x [Sτ

C(f

)]<

∞

Final words

Just as in linear systems theory, Lyapunov functionsprovide a characterization of system properties, as wellas a practical verification tool

Much is left out of this survey - in particular,

• Converse theory• Limit theory• Approximation techniques to construct Lyapunov functions or approximations to value functions• Application to controlled Markov processes, and approximate dynamic programming

π(f

)<

∞

DV (x) ≤ −f(x) + bIC(x)

‖P t (x, · ) − π‖f → 0

sup

CE

x [Sτ

C(f

)]<

∞

References

[1] S. P. Meyn and R. L. Tweedie.

[1,4] ψ-Irreducible foundations

[2,11,12,13] Mean-�eld models, ODE models, and Lyapunov functions

[1,4,5,9,10] Operator-theoretic methods. See also appendix of [2]

[3,6,7,10] Generators and continuous time models

Markov chains and stochasticstability. Cambridge University Press, Cambridge, secondedition, 2009. Published in the Cambridge MathematicalLibrary.

[2] S. P. Meyn. Control Techniques for Complex Networks. Cam-bridge University Press, Cambridge, 2007. Pre-publicationedition online: http://black.csl.uiuc.edu/˜meyn.

[3] S. N. Ethier and T. G. Kurtz. Markov Processes : Charac-terization and Convergence. John Wiley & Sons, New York,1986.

[4] E. Nummelin. General Irreducible Markov Chains and Non-negative Operators. Cambridge University Press, Cambridge,1984.

[5] S. P. Meyn and R. L. Tweedie. Generalized resolventsand Harris recurrence of Markov processes. ContemporaryMathematics, 149:227–250, 1993.

[6] S. P. Meyn and R. L. Tweedie. Stability of Markovianprocesses III: Foster-Lyapunov criteria for continuous timeprocesses. Adv. Appl. Probab., 25:518–548, 1993.

[7] D. Down, S. P. Meyn, and R. L. Tweedie. Exponentialand uniform ergodicity of Markov processes. Ann. Probab.,23(4):1671–1691, 1995.

[8] P. W. Glynn and S. P. Meyn. A Liapounov bound for solutionsof the Poisson equation. Ann. Probab., 24(2):916–931, 1996.

[9] I. Kontoyiannis and S. P. Meyn. Spectral theory and limittheorems for geometrically ergodic Markov processes. Ann.Appl. Probab., 13:304–362, 2003. Presented at the INFORMSApplied Probability Conference, NYC, July, 2001.

[10] I. Kontoyiannis and S. P. Meyn. Large deviations asymptoticsand the spectral theory of multiplicatively regular Markovprocesses. Electron. J. Probab., 10(3):61–123 (electronic),2005.

[11] W. Chen, D. Huang, A. Kulkarni, J. Unnikrishnan, Q. Zhu,P. Mehta, S. Meyn, and A. Wierman. Approximate dynamicprogramming using fluid and diffusion approximations withapplications to power management. Accepted for inclusion inthe 48th IEEE Conference on Decision and Control, December16-18 2009.

[12] P. Mehta and S. Meyn. Q-learning and Pontryagin’s MinimumPrinciple. Accepted for inclusion in the 48th IEEE Conferenceon Decision and Control, December 16-18 2009.

[13] G. Fort, S. Meyn, E. Moulines, and P. Priouret. ODEmethods for skip-free Markov chain stability with applicationsto MCMC. Ann. Appl. Probab., 18(2):664–707, 2008.

See also earlier seminal work by Hordijk, Tweedie, ... full references in [1].