OPTIMAL STABILIZATION USING LYAPUNOV …home.eng.iastate.edu/~ugvaidya/papers/Lyapunov_ocp.pdfOPTIMAL STABILIZATION USING LYAPUNOV MEASURE ARVIND U. RAGHUNATHAN ... Lyapunov measure,

OPTIMAL STABILIZATION USING LYAPUNOV MEASURE

ARVIND U. RAGHUNATHAN ∗ AND UMESH VAIDYA †

Abstract. Numerical solution for the optimal feedback stabilization of discrete time dynamicalsystems is the focus of the paper. Set-theoretic notion of almost everywhere stability introducedby the Lyapunov measure, which is weaker than conventional Lyapunov function based stabilizationmethods, is used for the optimal stabilization. The linear Perron-Frobenius transfer operator is usedto pose the optimal stabilization problem as an infinite dimensional linear program. Set-orientednumerical methods are used to obtain the finite dimensional approximation of the linear program. Weprovide conditions for existence of stabilizing feedback controls and for obtaining optimal stabilizingfeedback control as a solution of finite dimensional linear program. The approach is demonstrated ontwo examples. We also provide insights into the the connection between the optimal finite dimensionalfeedback controller and the stability of the inifinite dimensional problem in terms of the eigenvaluegap in the Markov matrix corresponding to the finite dimensional approximation.

Key words. Almost everywhere stability, nonlinear stabilization, Lyapunov measure

1. Introduction. Stability analysis and stabilization of nonlinear systems aretwo of the most important and extensively studied problems in control theory. Lya-punov function and Lyapuov function based methods have played a important rolein providing solutions to these problems. In particular Lyapunov function is usedfor stability analysis and control Lyapunov function (CLF) is used in the design ofstabilizing feedback controllers. Under the assumption that control Lyapunov func-tion exists stabilizing controller can be obtained using Sontag formula [1]. Howeverit is not easy to find the CLF and there is no systematic procedure to construct one.On the other hand in many application we are not just interested in stabilizationbut also in improving performance by minimizing a relevant cost function. Designingcontroller to minimize a given cost function is the objective of the optimal controlproblem (OCP). Optimal control for the OCP can be obtained from the solution ofHamilton Jacobi Bellman (HJB) equation. Under the additional assumption of de-tectability and stabilizability of nonlinear system, the optimal cost function if positivecan also be used as control Lyapunov function thus establishing the connection be-tween stability (Lyapunov function) and optimality (HJB equation). HJB equation isa nonlinear partial differential equation and hence difficult to solve analytically andone has to resort to approximate numerical scheme for its solution. Given the impor-tance of optimal control problem in various application, there is extensive literatureon numerical approximation of the HJB equation. We review some of the literatureparticularly relevant to this paper on the approximation of HJB equation and OCP.

In [2], adaptive space discretization scheme is used to obtain the solution of deter-ministic and stochastic discrete time HJB (dynamic programming) equation. Optimalcost function is obtained as a fixed point solution of linear dynamic programming op-erator. In [3, 4], cell mapping approach is used to construct an approximate numericalsolution for the deterministic and stochastic solution to optimal control problem. Thismethod essentially relies on replacing the point to point control evolution with theevolution of finitely many cells obtained from the discretization of the state space. In[5, 6], set oriented numerical methods based are used to underestimate the optimalone-step cost for transition between different sets in the context of optimal control and

∗United Technologies Research Center, East Hartford, CT 06108.([email protected]).†Dept. of Elec. and Comp. Engg., Iowa State University, Ames, IA-50011

([email protected]).

1

2 ARVIND RAGHUNATHAN AND UMESH VAIDYA

optimal stabilization problems respectively. This allows to represent the minimal costcontrol problem as one of finding the minimum cost path to reach the invariant set ona graph with edge weights derived from the under-estimation procedure. Dijkstra’salgorithm is used to construct approximate solution to HJB equation. In [7, 8, 9]solutions to stochastic and deterministic optimal control problems are proposed usinglinear programming approach. Optimal control problem is first cast as an infinitedimensional linear program. Approximate solution to the infinite dimensional linearprogram is then obtained using finite dimensional approximation of the linear pro-gramming problems or using sequence of LMI relaxation. The work in this paper alsodraws some connection to the research on optimization and stabilization of controlledMarkov chains [10].

In this paper we propose the use of Lyapunov measure for the optimal stabiliza-tion of nonlinear systems. Lyapunov measure is introduced in [11], to study weakerset wise notion of almost everywhere stability and is shown to be dual to Lyapunovfunction. Existence of Lyapunov measure guarantees stability from almost every withrespect to Lebesgue measure initial conditions in the phase space. The notion of al-most everywhere stability and density function to verify it was introduced by Rantzer[12]. Density function is also used in the design of almost everywhere stabilizing feed-back controller [13]. Just like control Lyapunov function is used in the Lyapunovfunction based feedback stabilization of nonlinear system, control Lyapunov measureis introduced in [14] to provide Lyapunov measure based framework for the feedbackstabilization of nonlinear systems. The co-design problem of jointly obtaining controlLyapunov measure and the stabilizing controller is posed as an infinite dimensionallinear program. Computational framework based on set oriented numerical meth-ods, developed by Dellnitz and group [15, 16], is proposed for the finite dimensionalapproximation of the linear program. In this paper we extend the Lyapunov mea-sure based framework and use it for the optimal stabilization of nonlinear system.The goal is to optimally stabilize an invariant set starting from almost every withrespect to Lebesgue measure initial conditions in the state space. The optimal sta-bilization problem using Lyapunov measure is posed as a infinite dimensional linearprogram. Computational framework based on the set oriented method is used for thefinite dimensional approximation of the linear program. Important feature of the pro-posed solution for the finite dimensional linear program is that deterministic feedbackcontrol solution is obtained for the optimal stabilization problem. Furthermore thesolution to the problem of designing deterministic feedback controller for stabilizationvia control Lyapunov measure, as addressed in [14], can be obtained as the specialcase of proposed solution to the optimal stabilization problem.

While infinite dimensional Lyapunov measure allows for the existence of onlyunstable dynamics in the complement of an invariant set, its finite dimensional ap-proximation allows for the existence of stable region with small domain of attractionin the complement of an invariant set. The “small” here refers to the size of thequantization of the state space. This further weaker notion of stability arise as a con-sequence of the discretization and is referred to as coarse stability in [11]. Althoughinsignificant from the point of view of any meaningful optimization ideally one wouldlike to avoid such coarse stable behavior. In this paper, we draw attention to thedesign of separation between first and second eigenvalues of the closed-loop Markovmatrix of the finite dimensional system in order to avoid such coarse behavior. Theseparation between the eigenvalues is guaranteed by the choice of the geometric decayparameter, γ > 1. This represents a departure from previous approaches for dis-

OPTIMAL STABILIZATION USING LYAPUNOV MEASURE 3

counted cost optimization algorithms where γ < 1. Based on our extensive numericalsimulation we provide necessary condition for the avoidance of such coarse behavior.

Lyapunov measure offers unique advantages for the problem of stabilization andcontrol. The controller designed using control Lyapunov measure allows for the exis-tence of unstable dynamics in the complement of the stabilized invariant set therebyexploiting the natural dynamics of the system. Exploiting natural dynamics for con-trol has been the theme of several research papers [17, 18, 19] including the celebratedOtto-Grebogi-Yorke (OGY) method for the control of chaos [20, 21]. Control designusing Lyapunov measure provides a systematic approach for exploiting the naturaldynamics of the system. Lyapunov measure also serves as a powerful tool for problemsinvolving control of complex non-equilibrium behavior. Complex non-equilibrium be-havior can be more naturally described on sets as opposed to on points and hencerequire measure theoretic tool for its analysis and control.

This paper is organized as follows. In section 2, we provide a brief overviewof some of the key result from [11, 22, 14] for stability analysis and stabilizationof nonlinear systems using Lyapunov measure. Framework for optimal stabilizationusing Lyapunov measure and transfer operators is posed as an infinite dimensionallinear program in section 3. Computation approach based on set oriented numericalmethods in proposed for the finite dimensional approximation of the linear program insection 4. Simulation result for optimal stabilization of pendulum on a cart exampleand one dimensional logistic map are presented in section 5. Connections between thecontroller obtained from finite dimensional approximation and the stability of infinitedimensional state-space system are discussed in section 6. Conclusion and discussionfollows in section 7.

2. Lyapunov measure, stability and stabilization. Lyapunov measure andcontrol Lyapunov measure equation were introduced in [11, 22, 14], for stability verifi-cation and for stabilizing controller design of an attractor set in nonlinear dynamicalsystems. An attractor set A for the dynamical system T : X → X, where X ⊂ Rn

assumed to be compact, is defined as follows:Definition 2.1 (Attractor set). A closed set A is said to be T invariant if

T (A) = A. A closed T invariant set A is called an attractor set if there exists a localneighborhood V of A such that T (V ) ⊂ V .

Stability and stabilization problems for the attractor set A is studied using thefollowing set-theoretic notion of almost everywhere stability.

Definition 2.2 (Almost everywhere (a.e.) uniformly stable). An attractor setA is said to be almost everywhere uniformly stable with respect to finite measure m iffor every ε > 0, there exists an N(ε) < ∞ such that

∞∑n=N(ε)

m{x ∈ Ac : Tn(x) ∈ B} < ε

for every set B ⊂ X \ Uδ, where Uδ is the δ neighborhood of an invariant set A. Astronger notion of almost everywhere stability is defined as follows:

Definition 2.3 (Almost everywhere stable with geometric decay). An attractorset A is said to be almost everywhere stable with geometric decay with respect to finitemeasure m if given δ > 0, there exists K(δ) < ∞ and β < 1 such that

m{x ∈ Ac : Tn(x) ∈ B} < K(δ)βn ∀n ≥ 0

for all sets B ⊂ X \ Uδ, where Uδ is the δ-neighborhood of an attractor set A.


These set-theoretic notions of almost everywhere stability were studied using alinear transfer operator-based framework. Linear Perron-Frobenius (P-F) operator isused to provide verifiable condition for the almost everywhere stability of dynamicalsystems. The evolution of points as described by the dynamical system is oftencomplex and nonlinear, however the evolution of the sets (or the measure supportedon the set) as described by the P-F operator is linear. For any given continuousmapping T : X → X, P-F operator, denoted by PT : M(X) →M(X) is given by

PT [µ](B) =∫

X

χB(T (x))dµ(x) (2.1)

whereM(X) is the vector space of all measures supported on X, χB(x) is the indicatorfunction supported on the set B ⊂ B(X), and B(X) is the Borel sigma-algebra of X.For more details on the P-F operator refer to [23]. Since the stability property ofan invariant set in definition (2.2) is stated in terms of the transient behavior of thesystem on the complement of an invariant set Ac, we define sub-stochastic Markovoperator as a restriction of the P-F operator on the complement of the invariant setas follows:

P1T [µ](B) :=

∫Ac

χB(T (x))dµ(x) (2.2)

for any set B ∈ B(Ac) and µ ∈M(Ac). Condition for the almost everywhere stabilityof an attractor set A with respect to some finite measure m is defined in terms of theexistence of Lyapunov measure. The Lyapunov measure µ is defined as follows:

Definition 2.4 (Lyapunov measure). The Lyapunov measure is defined as anynon-negative measure µ, which is finite outside the δ neighborhood Uδ of an attractorset A and satisfies the following inequality

[P1T µ](B) < βµ(B) (2.3)

for some β ≤ 1 and all set B ⊂ X \ Uδ such that m(B) > 0.The Lyapunov measure equation is introduced in [22] to provide necessary and

sufficient condition for almost everywhere uniform stability of an attractor set A withrespect to any finite measure m. The Lyapunov measure µ is obtained as the non-negative solution of the following Lyapunov measure equation:

γP1T µ(B)− µ(B) = −m(B) (2.4)

for some γ ≥ 1. The precise theorems for almost everywhere uniform stability (Def.2.2) and a.e. stability with geometric decay (Def. 2.3) as proved in [22] is as follows:

Theorem 2.5. An attractor set A for the dynamical system T : X → X isalmost everywhere uniformly stable (a.e. stable with geometric decay) with respect tofinite measure m if and only if there exists a non-negative measure µ which is finiteon B(X \ Uδ) and satisfies

γP1T µ(B)− µ(B) = −m(B)

for any set B ⊂ X \ Uδ, where Uδ is the δ neighborhood of the invariant set A andγ = 1(γ > 1). The measure m is absolutely continuous with respect to the Lyapunovmeasure µ.


Typically the measure m in the Lyapunov measure equation (2.4) is taken to bethe Lebesgue measure. Stability of an invariant set with respect to Lebesgue almostevery initial condition starting from a given set S can be studied by taking m = mS

in the Lyapunov measure equation, where mS is the Lebesgue measure supported onthe set S.

Remark 2.6. In the subsequent section we use the notation m for the Lebesguemeasure, mS for the Lebesgue measure supported on set S and Uδ for the δ neighbor-hood of an invariant set A for a given fixed δ > 0.

One of the advantages of using the linear P-F operator for the stability anal-ysis is that efficient computation methods are available for the finite dimensionalapproximation of the P-F operator, which then can be used for the finite dimensionalapproximation of the Lyapunov measure. The finite dimensional approximation ofthe P-F operator is based on the set-oriented numerical scheme developed by Dellnitzand group [15], where one constructs a partition of the state space X denoted by Xand the P-F approximation [P ]ij as follows:

X = {D1, ..., DL} (2.5)

[P ]ij =m(T−1(Dj) ∩Di)

m(Di)(2.6)

where ∪Ni=1Di = X and m is the Lebesgue measure. The finite dimensional approx-

imation of the P-F operator arises as a Markov matrix P on the finite partition X .The support of the attractor set A is captured using the left eigenvector associatedwith eigenvalue one of the Markov matrix P . The restriction of P on the comple-ment of an attractor set A is denoted by a sub-Markov matrix P1 and is used in theconstruction of the finite dimensional approximation of the Lyapunov measure. Theexistence of the finite dimensional Lyapunov measure leads to further weaker notionof a.e. stability which is referred to as coarse stability in [11] and is defined as follows:

Definition 2.7 (Coarse stability). Consider an attractor set A ⊂ X0 togetherwith a finite partition X1 = {D1, ..., DL} of the complement set X1 = X \ X0. Theattractor set A is said to be coarse stable w.r.t a.e initial conditions in X1 if for anattractor set B ⊂ U ⊂ X1, there exists no sub-partition L = {Ds1 , Ds2 , ..., Dsl

} in X1

with domain S = ∪lk=1Dsk

such that B ⊂ S ⊂ U and T (S) ⊂ S.Connection between the a.e. stability concluded from the existence of the infinite

dimensional Lyapunov measure µ and the coarse stability result obtained using thefinite dimensional approximation of the Lyapunov measure can be obtained as follows:

Theorem 2.8. Suppose A is an attractor set in A ⊂ X0 ⊂ X with finite di-mensional invariant measure supported on the finite partition X0 of X0. Let thesub-Markov matrix P1 be the finite dimensional approximation of sub-Markov oper-ator P1

T constructed on the finite partition X1 of the complement set X1 = X \ X0.Suppose

1. A Lyapunov measure µ exists such that

P1T µ(B) < βµ(B)

for all B ⊂ X1 for some β ≤ 1 and µ ≈ m (equivalent measures). Then thefinite dimensional approximation P1 is transient i.e., [Pn

1 ]ij → 0 as n → ∞and for all i, j = 1, ..., L.

2. P1 is transient then A is coarse stable with respect to the initial conditions inX1.


In other words, a.e. stability implies that the sub-Markov matrix P1 is transientand the transience of P1 implies the coarse stability of the attractor set A. We referto [11], for the discussion on several equivalent ways for checking the transient natureof the sub-Markov matrix P1 to ensure coarse stability of an attractor set.

In [14], control Lyapunov measure is introduced for the design of stabilizing feed-back controller. For the stabilization problem we consider the control dynamicalsystem of the form

xn+1 = T (xn, un)

where xn ∈ X ⊂ Rn and un ∈ U ⊂ Rd is the state space and the control spacerespectively. Both X and U are assumed to be compact. The objective is to designa feedback controller un = K(xn) to stabilize the attractor set A. The stabilizationproblem is solved using the Lyapunov measure by extending the P-F operator for-malism to the control dynamical system as follows. We define the feedback controlmapping C : X → Y := X × U as C(x) = (x,K(x)). Using the definition of thefeedback controller mapping C, we write the feedback control system as

xn+1 = T (xn,K(xn)) = T ◦ C(xn)

The system mapping T : Y → X and the control mapping C : X → Y can beassociated with P-F operators PT : M(Y ) → M(X) and PC : M(X) → M(Y )respectively and are defined as follows

PT [θ](B) =∫

Y

χB(T (y))dθ(y)

PC [µ](D) =∫

D

f(a|x)dm(a)dµ(x)

where θ ∈M(Y ), µ ∈M(X) and B ⊂ X, D ⊂ Y . f(a|x) is the conditional probabilitydensity function and is introduced to incorporate the particular form of feedbackcontroller mapping C(x) = (x,K(x)). The advantage of writing the feedback controldynamical system as the composition of two maps T : Y → X and C : X → Y is thatthe P-F operator for the composition T ◦ C : X → X can be written as a product ofPT and PC as follows:

PT◦C = PT · PC : M(X) →M(X) (2.7)

For the proof of the equality (2.7), please refer to [14]. Just like Lyapunov measureis used for the stability analysis of the attractor set for the autonomous system.Control Lyapunov measure is introduced to solve the stabilization problem. ControlLyapunov measure is defined as any non-negative measure µ ∈M(Ac), which is finiteon B(X \ Uδ) and satisfies

P1T · P1

C µ(B) < βµ(B) (2.8)

for every set B ⊂ X \Uδ and β < 1. P1T and P1

C are the restriction of the P-F operatorPT and PC respectively to the complement of the invariant set Ac and are definedsimilar to the restriction of the P-F operator in the autonomous case in equation (2.2).Stabilization of invariant set is posed as a co-design problem of jointly obtaining the


control Lyapunov measure µ and the control P-F operator PC or in particular theconditional probability density function f(a|x). After suitable change of coordinatesthe co-design problem is formulated as an infinite dimensional linear program. Com-putational method based on the set oriented numerical approach and switched systemformalism is proposed for the finite dimensional approximation of the linear program.The finite linear program is used to obtain the finite dimensional approximation ofP1

C denoted by P 1C and the control Lyapunov measure. Given the stochastic nature

of P 1T , the solution of finite linear program typically leads to the stochastic nature

of control matrix P 1C and hence stochastic control. To obtain deterministic control,

integer constraints on the entries of the matrix P 1C needs to be imposed which requires

solving mixed integer linear program.The purpose of this paper and the following sections is to extend the Lyapunov

measure based framework for the optimal stabilization of a nonlinear systems. Oneof the key highlights of this paper is that the deterministic finite optimal stabilizingcontrol is obtained as the solution of finite linear program. The finite deterministicstabilizing control is obtained as a special case of optimal stabilizing control, whenthe cost function is constant independent of the states and the control.

3. Optimal stabilization. The objective is to design a feedback controller forthe stabilization of the invariant set A in a.e. sense while minimizing the cost function.Consider the following discrete time control dynamical system

xn+1 = T (xn, un) (3.1)

where xn ∈ X ⊂ Rn and un ∈ U ⊂ Rd are state and control input respectively. Thestate space X and the control space U are assumed to be compact. Let Y := X×U andX1 := X\Uδ, where Uδ is the δ neighborhood of an invariant set A for some fixed δ > 0.We assume that there exists an feedback controller map C0(x) = (x,K0(x)), whichlocally stabilize the invariant set A i.e., there exists a neighborhood U of A such thatT ◦C0(U) ⊂ U and xn → A for all x0 ∈ U moreover A ⊂ Uδ ⊂ U . Typically controllermapping C0(x) can be obtained from linearization of the nonlinear mapping T in theneighborhood of the invariant set A. The task is to construct the optimal stabilizingcontroller for almost every initial conditions starting from X1. Let C1 : X1 → Y bethe stabilizing control, the union of the two control maps C0 and C1 is denoted byC : X → Y . So the global control map C is assumed to have a locally stabilizingcontroller map C0 embedded in it. Furthermore we assume that the feedback controlsystem T ◦ C : X → X is non-singular with respect to Lebesgue measure m, i.e.,m((T ◦ C)−1(B)) = 0 for all set B such that m(B) = 0. The cost of stabilization ofthe invariant set A with respect to initial conditions starting from the set B ⊂ X1 isdenoted by C(B) and is given by the following formula

C(B) =∫

B

∞∑n=0

γnG(xn, un)dm(x) (3.2)

where γ ≥ 1, x0 = x, xn+1 = T (xn, un) for n ≥ 0, and the cost function G : Y → R isassumed to be continuous non-negative and real valued function such that G(A, 0) = 0.We assume that the cost function G be such that (3.2) is finite for any set B ⊂ X1 andfor γ = 1. The parameter γ plays an important role in deciding the degree of stabilityof the invariant set A. We are interested in designing a feedback controller; hence fora given stabilizing feedback controller mapping C(x) = (x,K(x)), the stabilization


cost CC(B) is given as follows:

CC(B) =∫

B

∞∑n=0

γnG ◦ C(xn)dm(x) where x0 = x. (3.3)

Note that the global cost of stabilization of the invariant set A is the special case ofB = X1. Since we assume that γ ≥ 1; we need to impose stronger conditions on thecontroller mapping C for (3.3) to be finite. In particular, for γ > 1, we require thatthe controller map C stabilize the attractor set A with geometric decay rate β < 1

γ

(definition (2.3)) and for γ = 1 the invariant set A is a.e. uniformly stable (definition2.2). In the following theorem we show that the cost of stabilization of an invariantset A as given in equation (3.3) can be expressed using the control Lyapunov measureµ.

Theorem 3.1. Let γ ≥ 1 in the cost function (3.2) and the controller mappingC(x) = (x, K(x)) is designed such that the invariant set A is a.e. stable with geometricdecay rate β < 1

γ . The cost of stabilization of an invariant set A with respect toLebesgue a.e initial conditions starting from a set B can be expressed as follows:

CC(B) =∫

B

∞∑n=0

γnG(xn, un)dm(x) =∫

Ac×U

G(y)d[P1C µB ](y) =

⟨G, P1

C µB

⟩Ac×U

(3.4)where µB is the solution of the following control Lyapunov measure equation

γP1T · P1

C µB(D)− µB(D) = −mB(D) (3.5)

for every set D ⊂ X1, where mB is the Lebesgue measure supported on the set B ⊂X1.

We provide the proof of this theorem at the end of this section.Remark 3.2. By appropriately selecting the measure on the right hand side of

the Lyapunov measure equation (2.4) or the control Lyapunov measure equation (3.5),stability and stabilization of the invariant set with respect to a.e. initial conditionsstarting from different initial sets can be studied.

The minimum cost of stabilization is defined as the minimum over all a.e. stabi-lizing controller mapping C as follows:

C∗(B) = minC

CC(B) (3.6)

Defining θ(O) := [P1Cµ](O) for any set O ⊂ X1×U , θ ∈M(Ac×U), µ ∈M(Ac),

the inner product 〈f, µ〉X :=∫

Xfdµ(x) and using (3.5) (3.4), the optimal stabilization

problem of an invariant set A with respect to initial conditions starting from the setB can be posed as following infinite dimensional linear program.

minθ≥0

〈G, θ〉Ac×U (3.7a)

s.t. γ[P1T θ](D)− µ(D) = −mB(D) ∀D ⊂ X1 (3.7b)

∫D×U

dθ(y) = µ(D) ∀D ⊂ X1 (3.7c)


for some γ ≥ 1. In deriving the last inequality for the above linear program we haveused the fact that

θ(O) = [P1Cµ](O) =

∫O

f(a|x)dm(a)dµ(x)

and that f(a|x) is the conditional probability density. This infinite dimensional linearprogram need to be solved for the unknown measure θ and µ. We write the abovelinear program in slightly different form, to get rid of the unknown measure µ, since itcan be derived from θ. To do this we first define a projection map P1 : Ac × U → Ac

as follows:

P1(x, u) = x, P−11 (x) = (x, U)

P-F operator PP1 : M(Ac × U) →M(Ac) corresponding to the P1 can be written as

[P1P1

θ](D) =∫

Ac×U

χD(P1(y))dθ(y) =∫

D×U

dθ(y)

and

[P1P1

θ](D) =∫

D×U

dθ(y) = µ(D)

Using this definition of projection mapping P1 and the corresponding P-F operator,we can rewrite the linear program (3.7) with single unknown variable θ as follows:

minθ≥0

〈G, θ〉Ac×U (3.8a)

s.t. γ[P1T θ](D)− [P1

P1θ](D) = −mB(D) ∀D ⊂ X1 (3.8b)

Remark 3.3. Observe that the geometric decay parameter satisfies γ > 1. Thisis in departure with most optimization problems studied in the context of Markovcontrolled processes such as in Lasserre and Hernandez-Lerma [24]. Average cost anddicounted cost optimality problems are considered in [24]. We show in our section onnumerical computations (section 5) that the additional flexibility provided by γ ≥ 1 issufficient to guarantee that the finite dimensional controller also stabilizes the infinitedimensional system.

Our next goal is to derive the infinite dimensional dual to the above linear pro-gram. To do this we first define another linear transfer operator, which forms a dualto the P-F operator.

Definition 3.4 (Koopman operator). For a continuous mapping F : X1 → X2,Koopman operator UF : C0(X2) → C0(X1) is given by

(UF f)(x) = f(F (x))

where f ∈ C0(X2) and C0(Xi) is the space of all continuous function on Xi for i =1, 2.

With the mapping F : X1 → X2, one can also associate a P-F operator PF :M(X1) →M(X2) given by µ2(B) = [PF µ1](B) = µ1(F−1(B)), where µ1 ∈M(X1), µ2 ∈


M(X2) and B ⊂ X2. These two linear operators are dual to each other and the dualityis expressed using the following inner product

〈UF f, µ1〉X1=

∫X1

f(F (x1))dµ1(x1) =∫

X2

f(x2)d[PF µ1](x2) = 〈f, PF µ1〉X2(3.9)

Using the definition of the Koopman operator (3.4), we can associate Koopmanoperators UT : C0(X) → C0(Y ), UC : C0(Y ) → C0(X), and UP1 : C0(Ac) → C0(Ac×U)to the system T : Y → X, control map C : X → Y , and projection map P1 : Ac×U →Ac respectively. These three operators are defined as follows:

(UT f)(y) = f(T (y)) (3.10)

(UCg)(x) = g(C(x)) (3.11)

(UP1f)(y) = f(P1(y)) = f(x) (3.12)

where f ∈ C0(X) and g ∈ C0(Y ). We can define the restriction of the Koopmanoperators UT and UC to the complement of an invariant set i.e., on Ac

u := Ac × Uand Ac as follows:

U1T : C0(Ac) → C0(Ac

u), (U1T f)(y) = f(T (y))

U1C : C0(Ac) → C0(Ac

u), (U1Cg)(x) = g(C(x))

where f ∈ C0(Ac) and g ∈ C0(Acu). The duality between the restriction of Koop-

man and P-F operators U1 and P1 respectively can be expressed using the dualityrelation similar to (3.9). The Lagrange dual to the infinite dimensional linear program(3.7) can be written as follows

maxV

minθ≥0

〈G, θ〉Acu

+⟨V, γP1

T θ − P1P1

θ + mB

⟩Ac (3.13)

Rearranging terms we get

maxV

〈V,mB〉Ac + minθ≥0

⟨G + γU1

T V − U1P1

V, θ⟩

Acu

(3.14)

Hence the dual linear program is

minV

〈V,mB〉 (3.15a)

s.t. G(y) + γ(U1T V )(y)− (U1

P1V )(y) ≥ 0 (3.15b)

Now we provide the proof of theorem (3.1).Proof. We prove the theorem for the case when γ > 1. The proof for γ = 1 follows

along the same lines. For γ > 1, the controller mapping C : X → Y is assumed tobe stabilizing the invariant set A a.e. with geometric decay rate β < 1

γ . Hence thereexists an K(δ) < ∞ such that

m{x ∈ Ac : (T ◦ C)n(x) ∈ D} < Kβn


for every set D ⊂ X1. Now using the result from theorem (2.5), we know that thereexists non-negative measure µ which is finite on B(X1) and satisfies

γ[P1T◦C µ](D)− µ(D) = −m(D)

and (2.7) we get

γ[P1T · P1

C µ](D)− µ(D) = −m(D)

For the cost of stabilization of a set B, we have

CC(B) =∫

B

∞∑n=0

γnG◦C(xn)dm(x) =∫

B

limN→∞

N∑n=0

γnG◦C(xn)dm(x) =∫

B

limN→∞

fN (x)dm(x) x0 = x

where fN (x) =∑N

n=0 γnG ◦ C(xn). Since G ≥ 0, fN (x) ≤ fN+1(x) and usingmonotone convergence theorem, we have∫

B

limN→∞

fN (x)dm(x) = limN→∞

∫Ac

fN (x)dmB(x) = limN→∞

N∑n=0

〈γnG ◦ C(xn),mB〉Ac

limN→∞

N∑n=0

〈γnG ◦ C(xn),mB〉Ac = limN→∞

⟨U1

CG,

N∑n=0

γn[P1T◦C ]nmB

⟩Ac

where we have used the fact that xn = (T ◦ C)n(x) and the duality between theKoopman operator U1

T◦C and the P-F operator P1T◦C . Let µN

B =∑N

n=0 γn[P1T◦C ]nmB .

The measure µNB is absolutely continuous with respect to Lebesgue measure m for all

N . This follows because for any set D ⊂ X1 if m(D) = 0 then ([P1T◦C ]nmB)(D) =

m((T ◦ C)−n(D) ∩ B) = 0 for all n and every set B ⊂ X1. The later is true becauseof the non-singularity assumption of the closed loop map T ◦ C. Moreover µN

B (D) ≤µN+1

B (D) for every set D,B ⊂ X1. Hence there exists an integrable function gN (x) ≥ 0such that gN (x) ≤ gN+1(x) and dµN

B (x) = gN (x)dm(x). So we have

limN→∞

⟨U1

CG,N∑

n=0

γn[P1T◦C ]nmB

⟩Ac

= limN→∞

∫Ac

(U1CG)(x)gN (x)dm(x)

=∫

Ac

(U1CG)(x) lim

N→∞gN (x)dm(x) =

⟨U1

CG, µB

⟩Ac

where µB :=∑∞

n=0 γn[P1T◦C ]nmB =

∑∞n=0 γn[PT · P1

C ]nmB ] and µB is known tofinite on any set D ⊂ X1 because of a.e. stability property of the invariant set A withgeometric decay rate β < 1

γ . Furthermore µB satisfies following control Lyapunovmeasure equation

γ[P1T · P1

C µB ](D)− µB(D) = −mB(D) (3.16)

for every set D ⊂ X \ Uδ. Finally using the duality between U1C and P1

C , we get⟨U1

CG, µB

⟩Ac =

⟨G, P1

C µB

⟩Au

c

In the next section, we propose a computation framework based on the set ori-ented numerical methods for the finite dimensional approximation of the optimalstabilization problem. Optimal control for stabilization is obtained using the finitedimensional approximation of the primal-dual formalism introduced in this section.


4. Computational approach. The objective of the present section is to presenta computational framework for the solution of the finite-dimensional approximation ofthe optimal stabilization problem in (3.7a)-(3.7c). There exist a number of referencesrelated to the solution of inifinte dimensional linear programs in general and thosearising from control of markov processes, few of which we describe in the following.The monograph by Anderson and Nash [25] is an excellent reference on the propertiesof infinte dimensional linear programs. Lasserre, Hernandez-Lerma and co-workers[24, 26, 27] have in particular studied the solution of the infinte dimensional linearprograms that arise from control of Markov processes. Much of the work by Lasserreand Hernandez-Lerma has focussed on the average cost and discounted cost optimalityproblems.

Our intent is not to solve the computationally prohibitive infinte-dimensionalproblem and we will not present any results in that regard. Instead our objective is touse the finite-dimensional approximation to provide strong guarantees on the inifinite-dimensional system. We will first derive conditions under which solutions to the finite-dimensional approximation exist provided that the infinite dimensional system can bestabilized. We will use the existence result to show that finite dimensional version ofthe inifinite linear program (3.7a)-(3.7c) can be solved and that there always exists adeterministic controller.

4.1. Problem set-up in finite dimensions. For the purposes of computations,the infinite-dimensional stochastic description is replaced by its finite-dimensionalapproximations. We assume a finite partition of state-space X, and denote it byXN := {D1, ..., Di, ..., DN}, together with the associated measure space RN . Weassume without loss of generality that the invariant set, A is contained in DN , thatis A ⊆ DN . Typically, the size of DN is determined by the region of stability of alocal controller that attempts to stabilize the neighborhood of the invariant set. Wealso assume that for two partitions XN := {D1, . . . , DN} and XN ′ := {E1, . . . , EN ′}we have DN = EN ′ . Similarly the control space U is quantized and the control inputis assumed to take only finitely many control values from the quantized set

UM = {u1, . . . , ua, . . . , uM}, (4.1)

In addition, we will utilize the concept of a sub-partition which we define below. Apartition XN ′ := {E1, . . . , EN ′} for some N ′ > N is called a sub-partition of XN if foreach Ei there exists unique j ∈ {1, ..., N} such that Ei ∩Dj 6= ∅ and Ei ∩Dk = ∅ forall k 6= j.

The partition for the joint space Y , denoted by YN×M = XN×UM has cardinalityN ·M and is identified with an associated vector space RN×M . We use the notationPTa ∈ RN×N to denote the finite dimensional counterpart of P-F operator PT resultingfrom fixing the control action on the state-space to ua, that is u(D) = ua for allD ⊂ XN . The entries of PTa

are calculated as:

(PTa)ij :=

m(T−1a (Dj) ∩Di)m(Di)

where m is the Lebesgue measure and Ta(·) := T (·, ua). Since T : Y → X, we havethat PTa

is a Markov matrix. Additionally, P 1Ta

: RN−1 → RN−1 will denote the finitedimensional counterpart of P-F operator restricted to the complement of the invariantset, P1

T with the control input is fixed to ua. It is easily seen that P 1Ta

consists of thefirst (N − 1) rows and columuns of PTa

.


With the above quantization of the control space and partition of the state space,the determination of the control u(x) ∈ U (or equivalently K(x)) for all x ∈ Ac hasnow been cast as a problem of choosing u(Di) ∈ UM for all sets Di ⊂ XN . The finitedimensional approximation of the minimum cost of stabilization (3.7) is equivalent tosolving the following finite-dimensional linear program:

minθa,µ≥0

M∑a=1

(Ga)′θa (4.2a)

s.t. γM∑

a=1

(PTa)′θa − µ = −m (4.2b)

M∑a=1

θa = µ (4.2c)

where we have used the notation (·)′ for the transpose, m ∈ RN−1 and mj ≥ 0denotes the discrete counterpart of the Lebesgue measure m(B) in (3.7b), Ga ∈ RN−1

and Gaj is the cost associated with using control action ua on set Dj , θa, µ ∈ RN−1

are respectively the discrete counter-parts of infinte-dimensonal measure quantities in(3.7b)-(3.7c).

In the linear program (4.2) we have not enforced the constraint

θaj > 0 for exactly one a ∈ {1, ...,M} (4.3)

for each j = 1, ..., (N − 1). The above constraint ensures that the control obtainedis deterministic. We prove in the following that a deterministic controller can alwaysbe obtained provided the linear program (4.2) has a solution. To this end, we intro-duce the dual linear program [28] associated with the linear program in (4.2). TheLagrangian dual to the linear program obtained on elimination of the variable µ (bysubstituting the relation (4.2c) in (4.2b)) is,

maxV

minθa≥0

{M∑

a=1(Ga)

′θa + V

′(

γM∑

a=1(P 1

Ta)′θa −

M∑a=1

θa + m

)}= max

Vm

′V + min

θa≥0

{M∑

a=1(θa)

′ (Ga + γP 1

TaV − V

)}.

(4.4)

The Lagrangian dual has a finite solution only when,

Ga + γP 1Ta

V − V ≥ 0 ∀a = 1, ...,M.

Hence, the dual to linear program in (4.2) is,

maxV

m′V (4.5a)

s.t. V ≤ γP 1Ta

V + Ga ∀a = 1, ...,M. (4.5b)

In the above linear program (4.5), V are the dual variables to the equality contraints(4.2b).


4.2. Existence of stabilizing controls for a partition. The first step inshowing the existence of solutions to the finite linear program (4.2) is proving theexistence of a feasible solution. In this section, we will show the feasibility of thefinite linear program (4.2) under the assumption that the invariant set A is almosteverywhere stable for the dynamical system T (·, ·) using the finite number of controlvalues, UM . We state the key assumption below for future reference.

Assumption 4.1 (Existence of a stable fine-partition). There exists a partition ofthe state-space XN ′ := {E1, . . . , Ei, . . . , EN ′} with N ′ sufficiently large and associatedcontrols u(Ei) ∈ UM such that the system is coarse stable.

Remark 4.2. Although we do not have a proof, intuition suggests that assumption4.1 is likely to hold true and is necessary for the existence of the finite dimensionalcontroller.

In the following we show stability of the partition XN using stability of its sub-partition XN ′ . Key to this is the Markov chain representation of the dynamics of thefinite-dimensional state-space system. One of the important concept we will use isthat of hitting time. We define hitting time as the smallest number of time-steps inwhich a transition from any set of the partition to the set containing invariant set ispossible.

Definition 4.3. Given a parition XN := {D1, . . . , DN} of the state-space Xwith invariant set A ⊆ DN and a prescribed choice of control action uN (Di) on eachset of the partition, the hitting time τ(Di; (XN , uN )) of a set Di is defined as,

τ(Di; (XN , uN )) := min{

n

∣∣∣∣ (PTu)ii1(PTu

)i1i2 . . . (PTu)in−2in−1(PTu

)in−1N > 0for some i1, . . . , in−1 ∈ 1, . . . , (N − 1)

}(4.6)

where PTu ∈ RN×N denotes the finite dimensional approximation of the P-F operator.We state the following result on the coarse stability of a partition using the

defintion above.Lemma 4.4. Let P 1

Tube the closed loop sub-Markov matrix constructed on the

partition XN using the finite dimension control uN . The sub-Markov matrix P 1Tu

istransient if and only if

τ(Di; (XN , uN )) < N ∀Di ∈ XN . (4.7)

Proof. Consider the only if part first. Let the partition XN be coarse stable withchoice of controls uN (Di). We first show that the hitting time is finite for all thesets in the partition. Suppose, there exists Di ∈ XN such that τ(Di; (XN , uN )) = ∞.Consequently, there exists a partition of the index set {1, ..., (N − 1)} = I1 ∪ I2 whereI1 and I2 are disjoint and for each j ∈ I1 the hitting time of the set Dj is finiteand for each j ∈ I2 the hitting time of the set Dj is infinite (in particular, i ∈ I2).The restriction of the finite dimensional P-F approximation to the complement of theinvariant set P 1

Tucan then be shown to have following structure,

P 1Tu

=[

P1 00 P2

]where P1 ∈ R|I1|×|I1| and P2 ∈ R|I2|×|I2|. Further, since the hitting time is inifinite forall j ∈ I2 we have that all row sums of the matrix P2 is 1. By the Perron-FrobeniusTheorem for non-negative matrices [29] this implies that ρ(P2) = 1. From the block-diagonal structure we have that ρ(P 1

Tu) = 1 which contradicts the assumption that


P 1Tu

is transient. Hence, we have that the hitting time for all sets is finite. Now weshow that the hitting time is less than N . For some set Di, let τ(Di; (XN , uN )) = nand the set of indices be i1, . . . , in−1 such that Pii1 . . . Pin−1N > 0. Suppose il = ikfor some 1 ≤ l < k ≤ n − 1 then, eliminating indices il+1, . . . , ik we obtain a shortersequence which satisfies Pii1 . . . Pil−1il

Pilik+1 . . . Pin−1N > 0. This contradicts theminimality in the definition of the hitting time (4.6). Thus for each set Di the indicesin the sequence i1, . . . , in−1 must all be distinct and hence, τ(Di; (XN , uN )) < N .

Consider the if part of the claim. Let, L := maxi τ(Di; (XN , uN )). Now, for anyinitial distribution supported on the complement of the invariant set, i.e. µ ∈ RN−1,µ ≥ 0, µ 6= 0, the evolution of the distribution after L transitions is given by µ′(P 1

Tu)L.

From the definition of hitting time (4.6) we have that for any initial distribution onthe complement of the invariant set there is a non-zero probability of entering theinvariant set after L transitions. Hence, we have that

N−1∑i=1

(µ′(P 1Tu

)L)i <N−1∑i=1

µi.

Since the above holds true for all initial distribution we must have that (P 1Tu

)nL → 0as n → ∞. Hence, the sub-Markov matrix P 1

Tuis transient which implies coarse

stability of the partition XN with the choice of controls uN (Di).We will now prove the main result on the existence of stabilizing controls for a

partition XN under Assumption 4.1.Theorem 4.5. Suppose Assumption 4.1 holds. Then, there exists stabilizing

controls for any partition XN := {D1, . . . , DN} of the state-space.Proof. From Assumption 4.1, we have that there exists a fine-enough partition

XN ′ := {E1, . . . , EN ′} that is stabilizable using the finite controls in UM . In particularwe assume that XN ′ is a sub-partition of XN . Denote by uN ′(Ei) the stabilizingcontrols on each of the sets Ei ∈ XN ′ . Since, XN ′ is a sub-partition of XN we have thatthere exist nonempty sets Si ⊂ {1, . . . , N ′} for i = 1, . . . , N such that Di = ∪j∈Si

Ej .Further, Si ∩ Sk = ∅ for all i 6= k.

Consider the following procedure for identifying the controls on sets of the parti-tion XN .

1. For each i = 1, ..., N , let bi := arg minj∈Siτ(Ej ; (XN ′ , uN ′)).

2. Let uN (Di) := uN ′(Ebi) for all i = 1, ..., N

We will show in the following that 0 ≤ τ(Di; (XN , uN )) < N for all i = 1, . . . , Nwhich by Lemma 4.4 ensures transient nature of P 1

Tuconstructed over the partition

XN .Prior to proving this, we will need some additional results relating the P-F ma-

trices of the partitions. Let, PTaand QTa

represent respectively the P-F matricescorresponding to the partition XN and XN ′ for a fixed control action ua on all sets ofthe partitions. We show that the P-F matrices are related. By definition,

(QTa)ij =

m(Ei ∩ T−1a (Ej))

m(Ei)i, j = 1, . . . , N ′

where Ta(·) := T (·, ua). The entries of P-F matrix PTa are defined as,

(PTa)ij := m(Di∩T−1a (Dj))

m(Di)

= m((Di∩X)∩T−1a (Dj∩X))

m(Di)

=m((Di∩(∪k∈Si

Ek))∩T−1a (Dj∩(∪l∈Sj

El)))

m(Di)


Since Ei are pair-wise disjoint we have that,

T−1a (Dj ∩ ∪l∈Sj

El) = {x ∈ X|Ta(x) ∈ (Dj ∩ ∪l∈SjEl)}

= {x ∈ X|Ta(x) ∈ ∪l∈Sj(Dj ∩ El)}

= ∪l∈Sj{x ∈ X|Ta(x) ∈ (Dj ∩ El)}= ∪l∈Sj T

−1a (Dj ∩ El).

Additionally, from the definition of a sub-partition we also have for a given j the setsT−1

a (Dj ∩ El) are pair-wise disjoint. Hence,

(PTa)ij =

∑k∈Si,l∈Sj

m((Di∩Ek)∩T−1a (Dj∩El))

m(Di)

=∑

k∈Si,l∈Sj

m(Ek∩T−1a (El))

m(Di)

=∑

k∈Si,l∈Sj

(QTa )klm(Ek)m(Di)

.

(4.8)

We are now ready to prove the claim of the theorem. Suppose that the choiceof controls uN is not stabilizing for the partition XN . Then, there exists Di ∈ XN

such that τ(Di; (XN , uN )) = ∞. From the choice of controls we have that uN (Di) =uN ′(Ebi

). Denote by ai the index of control in UM corresponding to uN ′(Ebi), that

is uN ′(Ebi) = uai ∈ UM . From the stability of the sub-partition XN ′ , we know that

τ(Ebi; (XN ′ , uN ′)) < N ′

and additionally we also have that there exists a non-empty set K ⊆ {1, . . . , N ′} suchthat

τ(Ek; (XN ′ , uN ′)) = τ(Ebi; (XN ′ , uN ′))− 1 and (QTai

)bik > 0 ∀k ∈ K.

Now we first eliminate the possibility that Ek ⊂ Di for all k ∈ K. This is not possiblesince it contradicts our choice of bi in Step 2 for choosing the controls on each set ofthe partition. Hence, there exists k ∈ K such that Ek ⊂ Dj for some j different fromi. Since (QTai

)bik > 0, we have from (4.8) that (PTai)ij > 0. As a result, we have two

possibilities:• τ(Dj ; (XN , uN )) < ∞. If this is the case, then our assumption on the hitting

time of set Di, τ(Di; (XN , uN )) = ∞ is contradicted and our claim is proved.• τ(Dj ; (XN , uN )) = ∞. We can repeat the argument we made for Di for the

set Dj . Note that bj ≤ bi − 1. Given the finiteness of N and N ′ the abovelogic will terminate with bj = 0 which implies that a set Dj with a non-zeroone-step probability of transition to DN has been identified and the claim isproved.

4.3. Existence of solutions to finite linear program. In this section, weshow that existence of optimal deterministic controls solutions to the finite linearprogram (4.2). For the sake of simplicity and clarity of presentation, we will assumethat m > 0 in the following. The results presented here can easily be extended tothe case where m ≥ 0,m 6= 0. We will first derive conditions under which the linearprogram (4.2) is feasible.

Lemma 4.6. Suppose that Assumption (4.1) holds and m > 0. Then, for anypartition XN there exists γ > 1 such that for all γ ∈ [1, γ) there exists a feasiblesolution to the linear program (4.2).

Proof. From Theorem 4.5, we have that there exist stabilizing controls for thepartition XN . Hence, there exists a choice of controls uN (Di) on each cell of the


partition such that the P-F matrix for the resulting system denoted by PTu∈ RN×N

satisfies, ρ(P 1Tu

) < 1. Hence, there exists γ > 1 such that ρ(P 1Tu

) = 1/γ. Define,

θai :=

{xi if a : ua

i = uN (Di)0 otherwise. ∀i = 1, . . . , (N − 1).

Note that with the above definition, the constraints of linear program (4.2) can berecast as

(IN−1 − γ(P 1Tu

)′)x = m

where IN−1 ∈ R(N−1)×(N−1) is the identity matrix. Clearly, ρ(γP 1Tu

) < 1 for allγ ∈ [1, γ) and as a result the following holds,

x = (IN−1 − γ(P 1Tu

)′)−1m =

∞∑n=0

((γP 1Tu

)′)nm > 0.

This proves the claim.Remark 4.7. Lemma 4.6 has used the stabilizing controls identified in Theorem

4.5 to prove the existence of γ. This choice of γ might be conservative and in fact,there may be other choices of controls which allow feasibility of (4.2) for larger valuesof γ.

In the following we will derive conditions under which a solution to linear pro-gram (4.2) exists and then, show that condition for deterministic control (4.3) can besatisfied under the assumption of feasibility of linear program (4.2). The main resultis stated in Theorem 4.11.

Lemma 4.8. Suppose the Assumption 4.1 holds and m > 0, G(·, ·) ≥ 0 on thecomplement of the invariant set. Then for all γ ∈ [1, γ), there exists an optimalsolution θ to linear program (4.2) and an optimal solution V to the dual linear program

(4.5) with equal objective values (M∑

a=1(Ga)

′θa = m′V ).

Proof. Assumption 4.1 ensures that the linear program (4.2) is feasible (Lemma4.6). Observe that the linear program in (4.5) is always feasible with a choice ofV = 0. The claims hold as a result of linear programming strong duality [28].

Remark 4.9. Note that existence of an optimal solution does not impose positiv-ity requirement on the cost function G on the complement set. In fact, even assigningG(·, ·) = 0 allows determination of a stabilizing control from the Lyapunov measureequation (4.2b). In this case, any feasible solution to (4.2b)-(4.2c) is an optimalsolution and Theorem 4.5 guarantees the existence of such a solution.

The next result shows that linear program (4.2) always admits a determinsticcontrol action as an optimal solution. In the following, we will assume that the cost ispositive on the complement of the invariant set G(·, ·) > 0. This assumption is crucialin order to obtain deterministic controls.

Lemma 4.10. Suppose Assumption 4.1 holds and m > 0, G(·, ·) > 0 on thecomplement of the invariant set. Let θ solve (4.2) and V solve (4.5) for some γ ∈[1, γ). Then the following hold at the solution:

1. For each j = 1, ..., (N − 1) there exists at least one aj ∈ 1, ...,M such thatVj = γ(P 1

TajV )j + G

aj

j and θaj

j > 0 where Gaj

j := G(Dj , uaj )

2. There exists a θ that solves (4.2) and is such that for each j = 1, ..., (N − 1),there is exactly one aj ∈ 1, ...,M such that θ

aj

j > 0 and θa′

j = 0 for a′ 6= aj.


Proof. From the assumptions, we have that Lemma 4.8 holds. Hence, there exists(V, θ) that satisfy the first-order optimality conditions [28],

M∑a=1

θa − γM∑

a=1(P 1

Ta)′θa = m

V ≤ γP 1Ta

V + Ga ⊥ θa ≥ 0 ∀a = 1, ...,M.(4.9)

We will prove each of the claims in order.1. Suppose, there exists j ∈ 1, ..., (N − 1) such that θa

j = 0 for all a = 1, ...,M .Substituting in the optimality conditions (4.9) one obtains,

γM∑

a=1

((P 1Ta

)′θa)j = −mj

which cannot hold since, P 1Ta

has non-negative entries, γ > 0 and θa ≥ 0.Hence, there exists at least one aj such that θ

aj

j > 0. The complementaritycondition in (4.9) then requires that Vj = γ(P 1

TajV )j + G

aj

j . This proves thefirst claim.

2. Denote a(j) = min{a|θaj > 0} for each j = 1, ..., (N − 1). The existence of

a(j) for each j follows from statement 1. Define P 1Tu

∈ R(N−1)×(N−1) andGu ∈ RN−1 as follows:

(P 1Tu

)ji := (P 1Ta(j)

)ji ∀i = 1, ..., (N − 1)

Guj := G

a(j)j

(4.10)

for all j = 1, ..., (N − 1). From the definintion of P 1Tu

and Gu and comple-mentarity condition in (4.9) it is easily seen that V satisfies,

V = γP 1Tu

V + Gu = limn→∞

((γP 1Tu

)nV +n∑

k=0

(γP 1Tu

)kGu). (4.11)

Since V is bounded and Gu > 0 it follows that ρ(P 1Tu

) < 1/γ.Define θ as follows,

θa(1)1...

θa(N−1)N−1

= (IN−1 − γ(P 1Tu

)′)−1m (4.12a)

θaj = 0 ∀j = 1, ..., (N − 1), a 6= a(j). (4.12b)

The above is well-defined since we have already shown that ρ(P 1Tu

) < 1/γ.From the construction of θ, we have that for each j there exists only one aj ,namely a(j), for which θ

a(j)j > 0. It remains to show that θ solves (4.2). For

this observe that,

M∑a=1

(Ga)′θa =

N−1∑j=1

Ga(j)j θ

a(j)j

= (Gu)′(IN−1 − γ(P 1

Tu)′)−1m

= ((IN−1 − γP 1Tu

)−1Gu)′m

= V′m.

(4.13)


The primal and dual objectives are equal with above definition of θ and hence,θ solves (4.2). The claim is proved.

Lemma 4.10 shows that if there exists a solution to linear program (4.2) then, thethere exists a deterministic controller for the same. The following theorem states themain result.

Theorem 4.11. Given the system T : XN × UM → XN and m > 0, G(·, ·) > 0on the complement of the invariant set. If Assumption 4.1 holds, then the followingstatements hold for all γ ∈ [1, γ):

1. there exists a θ which is a solution to (4.2) and a V which is a solution to(4.5)

2. the optimal control for each set i = 1, ..., (N − 1) is given by,

u(Dj) = ua(j) (4.14)where a(j) := min{a|θa

j > 0}

3. µ satisfying

γ(P 1Tu

)′µ− µ = −m where (P 1

Tu)ji = (P 1

Ta(j))ji. (4.15)

is the Lyapunov measure for the controlled system.Proof. Assumption 4.1 ensures that the linear programs (4.2) and (4.5) have a

finite optimal solution (Lemma (4.8)). This proves the first claim of the theoremand also allows the applicability of Lemma 4.10. The remaining claims follow as aconsequence.

Remark 4.12. In [14], we proposed solving mixed integer linear program toobtain deterministic control solution to the feedback stabilization problem using controlLyapunov measure. However using the results of Theorem 4.11, one can solve alinear program, with cost function G = constant > 0, to obtain deterministic feedbackstabilizing control.

Though the results in this section have assumed that m > 0, this can be easilyrelaxed to m ≥ 0,m 6= 0. The case of m ≥ 0,m 6= 0 is of interest when the systemis not everywhere stabilizable. If it is known that there are regions of the state-spacethat are not stabilizable, then the m can be chosen such that its support is zero onthose regions. If the regions are not known a priori then, the (4.2) can be modified tominimize the l1-norm of the contraint residuals. This is similar to the feasibility phasethat is commonly employed in linear programming algorithms [30]. These and otherideas on computational complexity management will be addressed in a subsequentpaper.

5. Examples. In this section, we describe two numerical examples for the op-timal stabilization using Lyapunov measure using the computational framework de-veloped in the previous section. The results in this section have been obtained usingan interior-point algorithm, IPOPT [31]. IPOPT is an open-source software, availablethrough the COIN-OR repository (http://www.coin-or.org/), developed for solv-ing large-scale non-convex nonlinear programs. We recognize that IPOPT may notbe the most efficient algorithm for linear programs that arise in the context of thispaper. Our objective in this paper is to present some numerical evidence supportingthe claims in the paper. A more efficient algorithm that is tailored for the solution oflarger instances of the linear program will be presented in a subsequent paper.


5.1. Inverted pendulum on a cart. The problem of optimal stabilizationand optimal control for the inverted pendulum on cart using set oriented numericalmethods is discussed in [5, 6]. We borrow this example from [5, 6], but use Lyapunovmeasure for stabilization.

x1 = x2

x2 =gl sin(x1)− 1

2mrx22sin(2x1)− mr

mlcos(x1)u

43 −mrcos2(x1)

(5.1)

The parameters values that we use for simulation are M = 8,m = 2,mr = mm+M , l =

0.5, and g = 9.8. For uncontrolled system there are two equilibrium points: theequilibrium point at (π, 0) is stable in the Lyapunov sense with eigenvalues of thelinearization on the jω axis, the second equilibrium point at the origin is of type saddleand hence unstable. For the computational purpose the discrete time dynamicalsystem T is obtained from the continuous dynamics (5.1) by setting T (x0, y0) =φ(δt, x0, y0), where φ(t, x0, y0) is the solution of the differential equation (5.1) startingfrom initial condition (x0, y0). We use δt = 0.1. The state space X is chosen tobe [−π, π] × [−10, 10] and is partitioned into 302 boxes. The length of the each ofthese boxes is 2π

30 and 2030 along x and y direction respectively. The control u takes

finitely many values from the control set U = {−80,−70, ...,−10, 0, 10, ..., 70, 80}.The objective is to optimally stabilize the unstable equilibrium point at the origin,the upright vertical position of the pendulum. The cost function, which is used forthe optimization is

G(x, u) = x21 + x2

2 + u2

The value of γ used for the simulation is 1.05. Figure (5.1a), (5.1b) and (5.1c) showsthe phase portrait for the uncontrolled system, closed loop invariant measure, andthe eigenvalue plot for the closed loop Markov chain respectively. Figure (5.2a) and(5.2b) shows the plot of the optimal cost and the controller respectively. From thefigure (5.2a) we see that the optimal controller uses almost zero control along the stablemanifold of the uncontrolled equilibrium point at the origin thus exploiting the naturaldynamics of the system. Figure (5.2c) shows the fraction of initial conditions fromeach box that eventually end up at the origin. Figure (5.2c) is obtained by performinga time domain simulation for the closed loop system using the finite control and with30 initial conditions spread uniformly in each box of the partition. Importance andthe necessity of this plot will be explained in section (6). From this plot it is clearthat all the initial conditions eventually end up at the origin thus verifying the successof the finite dimensional control for stabilization.

5.2. Logistic map. Consider the following one dimensional control Logistic map

xn+1 = λxn(1− x2n) + un (5.2)

Logistic map is widely studied in dynamical system literature [32] for its complexdynamics. For the parameter value of λ approximately at 2.3 and beyond the Lo-gistic map is known to exhibit chaotic behavior through a cascade of period dou-bling bifurcation. Figure (5.3) shows the bifurcation diagram for the logistic mapand the invariant measure for the parameter value of λ = 2.3. For the controlof Logistic map, we let control u to take finitely many values from the control set


(a) (b) (c)

Fig. 5.1. (a)Phase portrait for the uncontrolled system (b) Invariant measure for the closedloop system (c) eigenvalues for the closed loop Markov matrix

(a) (b) (c)

Fig. 5.2. (a)Plot of the optimal control (b) Plot of the optimal cost (c) Fraction of initialconditions from each box that eventually end up at the origin

U = {−0.5,−0.45, ...,−0.05, 0, 0.05, 0.1, 0.15, ..., 0.5}. The control objective is to sta-bilize the unstable equilibrium point at the origin. The cost function used for theoptimization is

G(x, u) = u2

The value of γ used for the optimization is 1. Figure (5.3b) and (c) shows the plotsfor the open loop and closed loop invariant measure respectively. In figure (5.4), weplot the optimal control. Clearly the control has bang-bang characteristics outside thesmall neighborhood of the origin and is active only at few points in the state space.Figure (5.4c) shows the plot of fraction of initial conditions that eventually enteredthe origin from each cell of the finite dimension partition, validating that the originis stabilized with the finite control.

6. Closed loop stability with finite control. In this section we investigatethe stability of the closed loop feedback controlled system upon application of thefinite dimension control i.e., the stability of

xn+1 = T (xn,K(xn)) K(x) = ua(j) for x ∈ Dj & ua ∈ {u1, ...., uM} (6.1)

The existence of the finite dimensional control and the Lyapunov measure guaranteesthat the finite dimensional closed loop sub-Markov matrix P 1

Tuconstructed on the

complement set is transient. P 1Tu

is said to be transient if [(P 1Tu

)n]ij → 0 as n → 0for all i, j. However similar to the case of autonomous system (theorem 2.8); the


(a) (b) (c)

Fig. 5.3. (a)Bifurcation diagram for the logistic map (b) Invariant measure for the uncontrolledLogistic map (c) Invariant measure for the controlled logistic map

(a) (b) (c)

Fig. 5.4. (a) Optimal cost for the closed loop Logistic map (b) Plot of the optimal control (c)Fraction of initial conditions from each box that eventually end up at the origin

transient nature of P 1Tu

implies only the coarse stability of the attractor set for thefeedback control system.

Theorem 6.1. Assume that the closed loop sub-Markov matrix P 1Tu

constructedon the finite partition XN = {D1, ..., DN} and using finite control UM = {u1, ..., uM}is transient. Then the attractor set A for the feedback control system (6.1) is coarsestable.

Proof. Refer to [33].

Figure (6.1a) and (6.1b) show a typical stable behavior in the complement of theattractor set that is possible and not possible by the coarse stability of the attractorset respectively. The stable invariant set in the complement space is represented by x0

and their domain of attraction by Ds (small) and DL (large). In the ideal situationone would like to design the controller so as to avoid the presence of such coarsestable dynamics or atleast avoid the one with large domain of attraction DL. Fromour extensive numerical simulation we noticed that the presence of stable dynamicswith large domain of attraction DL are not completely unobservable and leave theirsignature on the finite dimensional closed loop Markov matrix. In particular the lackof enough separation between the eigenvalue at one and the second largest eigenvalueof the close loop Markov chain is indicative of the presence of stable dynamics withlarge domain of attraction in the complement set. To illustrate this point we consider


(a) (b)

Fig. 6.1. (a)Possible coarse stable set x0 with small (green) and large (red) domain of attractionDs & DL respectively (b) Stable regions not possible by coarse stability

the following example of damped pendulum on a cart from section (5.1).

x1 = x2

x2 =gl sin(x1)− 1

2mrx22sin(2x1)− mr

mlcos(x1)u

43 −mrcos2(x1)

− 2ζ

√g

lx2 (6.2)

The value of the damping constant ζ is taken to be 0.2 while the values of all otherparameters are same as in section (5.1). Figure (6.2) and (6.3) corresponds to thecase of controller design which leads to not enough separation between the first andthe second largest eigenvalue for the closed loop Markov matrix respectively. Theobjective is to stabilize the equilibrium point at the origin and the cost function Gfor both the case equal G(x, u) = x2

1 + x22 + u2.

(a) (b) (c)

Fig. 6.2. (a) Eigenvalues plot for closed loop Markov chain (b) Lyapunov measure (c) Fractionof initial conditions from each box that eventually end up at the origin

For both the cases the closed loop Markov matrix is transient and is confirmedfrom the existence of the Lyapunov measure as shown in figure (6.2b) and (6.3b).However figure (6.2c) shows that the true dynamics obtained after implementing thefinite dimensional controller is coarse stable with large domain of attraction. Figures(6.2c) and (6.3c) show the plot of fraction of initial conditions from each box of thepartition that eventually end up at the origin. These plot are obtained by performingtime domain simulation for closed loop system with finite control starting with uni-formly distributed 900 initial conditions from each box of the partition. From figure(6.2c), we see that only the points which lies in the cells at or near the stable manifold


(a) (b) (c)

Fig. 6.3. (a) Eigenvalues plot for closed loop Markov chain (b) Lyapunov measure (c) Fractionof initial conditions from each box that eventually end up at the origin

of the unstable fixed point at the origin end up eventually at the origin and all otherpoints do not converge to the origin. Comparing this scenario with figure (6.3c), wesee that points from every cells eventually end up at the origin. The important pointto notice is that the gap between the first and the second eigenvalue for the plot infigure (6.2a) is almost zero and equals 0.0012, whereas this gap is equal to 0.16 infigure (6.3c).

Existence of eigenvalue close to one for the finite dimensional Markov chain cor-responds to the presence of almost invariant region in the phase space [15]. Almostinvariant regions are the regions in the phase space where system trajectories spendlarge amount of time before finally entering the invariant region. For more details onalmost invariant region and its connection to the second eigenvalue of the Markov ma-trix refer to the [15, 34]. We argue that if there are stable regions with large domainof attraction in the complement of the stabilized invariant set and the closed loopsub-Markov matrix is transient then these stable regions appear as almost invariantregion for the finite dimensional approximation. We conjecture that for fine enoughpartition the lack of enough separation between the first and the second eigenvalueof the closed loop Markov chain is the necessary condition for the presence of stabledynamics with large domain of attraction in the complement of the stabilized invari-ant set. To illustrate this point consider the following example of the one dimensionalsystem:

xn+1 = xn − (xn − a1)(xn − b)(xn − a2), xn ∈ X = [0, 1]

This system has two stable equilibrium points at x = a1 and x = a2 and one unstableequilibrium point at x = b. We assume that 0 < a1 < 1

2 < b < a2. Consider thefollowing partition

X = {[0, b− ε], [b− ε, 1]}

where ε > 0 is a small parameter. Finite dimensional Markov matrix constructed onthe partition X is of the form

P =(

1 0ε 1− ε

)Clearly the eigenvalues of the Markov matrix are 1 and 1 − ε and the stable

region corresponding to the stable fixed point at x = a2 appears as almost invariant


region for the finite dimensional Markov matrix. In order to avoid the presence ofsuch coarse stable dynamics with large domain of attraction the finite dimensionalcontroller should be designed such that there is enough separation between the firstand the second eigenvalues of the closed loop Markov matrix. The gap betweenthe eigenvalues can be imposed by solving the finite dimensional linear program forvalues of γ greater than one if possible. The larger the value of γ, the more will bethe separation between the first and the second eigenvalues. Plots in figure (6.2) and(6.3) were obtained using value of γ equal to one and 1.05 respectively.

On the other hand stable regions with small domain of attraction Ds as shownin figure (6.1a), are unobservable and leave no signature on the closed loop Markovmatrix. Moreover we did not see the evidence of such behavior in our numerical sim-ulation. However the presence of such small stable region is not important from thepoint of view of any meaningful optimization as any small amount of noise or uncer-tainty will destroy such behavior. Rigorous connections between the finite dimensionalapproximation of the Markov chain and the true dynamics can be established usingthe stochastic stability results developed in [35, 36, 37], where the finite dimensionalMarkov matrix can be considered to be arising from the stochastic perturbation ofthe deterministic dynamical system. However such rigorous theoretical connectionsare developed only for lower dimensional hyperbolic maps. Detail investigations ofsuch results for the controlled dynamical systems are beyond the scope of this paperand will be the topic of future investigation.

7. Conclusions. Lyapunov measure is used for the optimal stabilization of anattractor set for a discrete time dynamical system. The optimal stabilization problemusing Lyapunov measure is posed as an infinite dimensional linear program. Com-putational framework based on set oriented numerical method is proposed for thefinite dimensional approximation of the linear program. Important feature of the pro-posed solution for the finite dimensional linear program is that deterministic feedbackcontrol is obtained for the optimal stabilization problem.

Set-theoretic notion of almost everywhere stability introduced by the Lyapunovmeasure offer several advantages for the problem of stabilization. First the controllerdesigned using Lyapunov measure exploits the natural dynamics of the system byallowing the existence of unstable dynamics in the complement of the stabilized set.The coarse stability introduced as a consequence of the finite dimensional approxi-mation of the Lyapunov measure takes the notion of a.e stability one step further byallowing the possibility for the existence of stable dynamics with small (size of thepartition) domain of attraction. Second, the Lyapunov measure provide systematicframework for the problem involving stabilization and control of system with complexnon-equilibrium behavior.

REFERENCES

[1] E. D. Sontag, “A “universal” construction of Artstein’s theorem on nonlinear stabilization,”Systems Control Lett., vol. 13, no. 2, pp. 117–123, 1989.

[2] L. Grune, “Error estimation and adaptive discretization for the discrete stochastic Hamilton-Jacobi-Bellman equation,” Numerische Mathematik, vol. 99, pp. 85–112, 2004.

[3] L. G. Crespo and J. Q. Sun, “Solution of fixed final state optimal control problem vis simplecell mapping,” Nonlinear dynamics, vol. 23, pp. 391–403, 2000.

[4] ——, “Stochastic optimal control via bellman principle,” Automatica, vol. 39, pp. 2109–2114,2003.

[5] O. Junge and H. Osinga, “A set oriented approach to global optimal control,” ESAIM: Control,Optimisation and Calculus of Variations, vol. 10, no. 2, pp. 259–270, 2004.


[6] L. Grune and O. Junge, “A set oriented approach to optimal feedback stabilization,” SystemsControl Lett., vol. 54, no. 2, pp. 169–180, 2005.

[7] D. Hernandez-Hernandez, O. Hernandez-Lerma, and M. Taksar, “A linear programming ap-proach to deterministic optimal control problems,” Applicationes Mathematicae, vol. 24,no. 1, pp. 17–33, 1996.

[8] V. Gaitsgory and S. Rossomakhine, “Linear programming approach to deterministic long runaverage optimal control problems,” SIAM J. Control ad Optimization, vol. 44, no. 6, pp.2006–2037, 2006.

[9] J. Lasserre, C. Prieur, and D. Henrion, “Nonlinear optimal control: Numerical approximationvia moment and lmi- relaxations,” in Proceeding of IEEE Conference on Decision andControl, Seville, Spain, 2005.

[10] S. Meyn, “Algorithm for optimization and stabilization of controlled Markov chains,” Sadhana,vol. 24, pp. 339–367, 1999.

[11] U. Vaidya and P. G. Mehta, “Lyapunov measure for almost everywhere stability,” IEEE Trans-actions on Automatic Control, 2007.

[12] A. Rantzer, “A dual to Lyapunov’s stability theorem,” Systems & Control Letters, vol. 42, pp.161–168, 2001.

[13] S. Prajna, P. A. Parrilo, and A. Rantzer, “Nonlinear control synthesis by convex optimization,”IEEE Transactions on Automatic Control, vol. 49, no. 2, pp. 1–5, 2004.

[14] U. Vaidya, P. Mehta, and U. Shanbhag, “Nonlinear stabilization using control lyapunovmeausre,” in Submitted to IEEE Conference on Decision and Control, New Orleans, LA,2007.

[15] M. Dellnitz and O. Junge, “On the approximation of complicated dynamical behavior,” SIAMJournal on Numerical Analysis, vol. 36, pp. 491–515, 1999.

[16] ——, Set oriented numerical methods for dynamical systems. World Scientific, 2000, pp.221–264.

[17] E. Bollt and J.D.Meiss, “Targeting chaotic orbit to the moon through recurrence,” Phys. Lett.A, vol. 204, pp. 373–378, 1995.

[18] W. Koon, M.W.Lo, and J. Marsden, “Low energy transfer to moon,” Celestical Mechanics,vol. 81, pp. 63–73, 2001.

[19] U. Vaidya and I. Mezic, “Controllability for a class of area preserving twist maps,” Physica D,vol. 189, pp. 234–246, 2004.

[20] E. Ott, C. Grebogi, and J. A. Yorke, “Controlling chaos,” Phys. Rev. Lett., vol. 64, no. 11, pp.1196–1199, 1990.

[21] T. Shinbrot, C. Grebogi, J. A. Yorke, and E. Ott, “Using small perturbations to control chaos,”Nature, vol. 363, pp. 411–417, June 1993.

[22] U. Vaidya, “Converse theorem for almost everywhere stability using Lyapunov measure,” inProceedings of American Control Conference, New York, NY, 2007.

[23] A. Lasota and M. C. Mackey, Chaos, Fractals, and Noise: Stochastic Aspects of Dynamics.New York: Springer-Verlag, 1994.

[24] O. Hernandez-Lerma and J. B. Lasserre, Discrete-time Markov Control Processes: Basic Op-timality Criteria. Springer-Verlag, New York, 1996.

[25] E. Anderson and P. Nash, Linear Programming in Infinite-Dimensional Spaces - Theory andApplications. John Wiley & Sons, Chichester, U.K., 1987.

[26] O. Hernandez-Lerma and J. B. Lasserre, “Approximation schemes for infinite linear programs,”SIAM J. Optimization, vol. 8, no. 4, pp. 973–988, 1998.

[27] O. Hernandez-Lerma and J. Gonzalez-Hernandez, “Constrained markov control processes inborel spaces: the discounted case,” Mathematics of Operations Research, vol. 52, pp. 271–285, 2000.

[28] O. L. Mangasarian, Nonlinear programming, ser. Classics in Applied Mathematics. Philadel-phia, PA: Society for Industrial and Applied Mathematics (SIAM), 1994, vol. 10, correctedreprint of the 1969 original.

[29] E. Seneta, Non-Negative Matrices. London: George Allen & Unwin Ltd, 1973.[30] S. J. Wright, Primal-Dual Interior-Point Methods. Philadelphia, Pa: Society for Industrial

and Applied Mathematics, 1997.[31] A. Wachter and L. T. Biegler, “On the implementaion of a primal-dual interior point filter line

search algorithm for large-scale nonlinear programming,” Mathematical Programming, vol.106, no. 1, pp. 25–57, 2006.

[32] A. Katok and B. Hasselblatt, Introduction to the modern theory of dynamical systems. Cam-bridge, UK: Cambridge University Press, 1995.

[33] U. Vaidya, P. Mehta, and U. Shanbhag, “Nonlinear stabilization via control lyapunov meausre,”in Submitted to IEEE Transactions on Automatic Control, New Orleans, LA, 2007.


[34] M. Dellnitz, O. Schtze, and T. Hestermeyer, “Covering pareto sets by multilevel subdivisiontechniques,” Journal of Optimization, Theory and Applications, vol. 124, no. 1, pp. 113–136, 2005.

[35] M. Blank and G. Keller, “Stochastic stability versus localization in one-dimensional chaoticdynamical systems,” Nonlinearity, vol. 10, pp. 81–107, 1997.

[36] A. Lasata and J. Yorke, “On the existence of invariant measure for piecewise monotone trans-formations,” Trans. Am. Math. Soc., vol. 186, pp. 481–8, 1973.

[37] Y. Kifer, Ergodic Theory of Random Transformations, ser. Progress of Probability and Statis-tics. Boston: Birkhauser, 1986, vol. 10.

Documents

OPTIMAL STABILIZATION USING LYAPUNOV …home.eng.iastate.edu/~ugvaidya/papers/Lyapunov_ocp.pdfOPTIMAL STABILIZATION USING LYAPUNOV MEASURE ARVIND U. RAGHUNATHAN ... Lyapunov measure,