418 IEEE TRANSACTIONS ON NEURAL NETWORKS AND …

418 IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, VOL. 25, NO. 2, FEBRUARY 2014

Decentralized Stabilization for a Class ofContinuous-Time Nonlinear Interconnected SystemsUsing Online Learning Optimal Control Approach

Derong Liu, Fellow, IEEE, Ding Wang, and Hongliang Li

Abstract— In this paper, using a neural-network-based onlinelearning optimal control approach, a novel decentralized con-trol strategy is developed to stabilize a class of continuous-time nonlinear interconnected large-scale systems. First, optimalcontrollers of the isolated subsystems are designed with costfunctions reflecting the bounds of interconnections. Then, it isproven that the decentralized control strategy of the overallsystem can be established by adding appropriate feedback gainsto the optimal control policies of the isolated subsystems. Next,an online policy iteration algorithm is presented to solve theHamilton–Jacobi–Bellman equations related to the optimal con-trol problem. Through constructing a set of critic neural net-works, the cost functions can be obtained approximately, followedby the control policies. Furthermore, the dynamics of the estima-tion errors of the critic networks are verified to be uniformly andultimately bounded. Finally, a simulation example is provided toillustrate the effectiveness of the present decentralized controlscheme.

Index Terms— Adaptive dynamic programming, decentralizedcontrol, large-scale systems, neural networks, nonlinearinterconnected systems, optimal control, policy iteration,reinforcement learning.

I. INTRODUCTION

VARIOUS complex systems in social and engineeringareas, such as ecosystems, transportation systems, and

power systems, are considered as large-scale systems. Gen-erally speaking, a large-scale system is comprised of severalsubsystems with obvious interconnections, which leads to theincreasing difficulty of analysis and synthesis when usingclassical centralized control techniques. Bakule [1] pointed outwith similar results that it is, therefore, necessary to partitionthe design issue of the overall system into manageable sub-problems. Then, the overall plant is no longer controlled bya single controller but by an array of independent controllersthat all together represent a decentralized controller. Therefore,the decentralized control has been a control of choice forlarge-scale systems because it is computationally efficient to

Manuscript received January 5, 2013; revised May 6, 2013;accepted July 25, 2013. Date of publication September 16, 2013; dateof current version January 10, 2014. This work was supported in part bythe National Natural Science Foundation of China under Grants 61034002,61233001, and 61273140, and in part by the Early Career DevelopmentAward of SKLMCCS. The acting Editor-in-Chief who handled the review ofthis paper was Danil Prokhorov.

The authors are with The State Key Laboratory of Management and Controlfor Complex Systems, Institute of Automation, Chinese Academy of Sciences,Beijing 100190, China (e-mail: [email protected]; [email protected];[email protected]).

Color versions of one or more of the figures in this paper are availableonline at http://ieeexplore.ieee.org.

Digital Object Identifier 10.1109/TNNLS.2013.2280013

formulate control law that use only locally available subsystemstates or outputs [2]. Actually, considerable attention has beenpaid to the decentralized stabilization of large-scale systemsduring the last several decades [3]–[7].

As previously mentioned, a decentralized strategy consistsof some noninteracting local controllers corresponding tothe isolated subsystems, not the overall system. Thus, inmany situations, the design of the isolated subsystems is amatter of great significance. In [8], it was shown that thedecentralized control of the interconnected system was relatedto the optimal control of the isolated subsystems. Therefore,the optimal control method can be employed to facilitate thedesign process of the decentralized control strategy. However,in [8], the cost functions of the isolated subsystems werenot chosen as the general forms, not to mention that thedetailed procedure was not given. For this reason, in thispaper, by employing the online policy iteration algorithm, wewill investigate the decentralized stabilization problem usingneural-network-based learning optimal control approach.

The optimal control of nonlinear system often leads tosolving the Hamilton–Jacobi–Bellman (HJB) equation insteadof the Riccati equation of the linear case. Though dynamicprogramming is a useful technique to solve the optimizationand optimal control problems, in many cases, it is com-putationally difficult to apply it because of the curse ofdimensionality. Fortunately, based on function approximators,such as neural networks, adaptive (or approximate) dynamicprogramming (ADP) was proposed by Werbos [9], [10] asan alternative method to solve the optimal control problemsforward-in-time. There are several synonyms used for ADP,including adaptive dynamic programming [11]–[15], approx-imate dynamic programming [16]–[18], neuro-dynamic pro-gramming [19], neural dynamic programming [20], adaptivecritic designs [21], and reinforcement learning [22]. In therecent years, great efforts have been made to ADP and relatedresearch in theory and applications. Numerous excellent resultshave been obtained that greatly promotes the development ofrelevant disciplines [23]–[40].

In light of [13], the ADP technique is closely relatedto reinforcement learning when engaging in the research offeedback control. In general, value and policy iterations arefundamental algorithms for the reinforcement learning-basedADP in optimal control. Policy iteration starts with a stabiliz-ing control, whereas value iteration cannot always guaranteethe stability of control during the implementation process.Al-Tamimi et al. [18], Zhang et al. [23], and Liu et al. [26]

2162-237X © 2013 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

LIU et al.: ONLINE LEARNING OPTIMAL CONTROL APPROACH 419

studied the optimal control problem of discrete-time non-linear systems using value iteration algorithm. Specifically,policy iteration represents a class of algorithms containing twobasic iterations, i.e., policy evaluation and policy improvement[41]–[46]. Abu-Khalaf and Lewis [41] derived an offlineoptimal control scheme for nonlinear systems with saturatingactuators. Then, Vrabie and Lewis [42] and Vamvoudakis andLewis [43] used online policy iteration algorithm to study theinfinite horizon optimal control of continuous-time nonlinearsystems, respectively. The former was performed based onthe sequential updates of two neural networks, namely, criticnetwork and action network, whereas in the latter, the twonetworks were trained simultaneously. Recently, Liu et al. [44]extended the policy iteration algorithm to nonlinear optimalcontrol problem with unknown internal dynamics and dis-counted cost function. Besides, Bhasin et al. [46] constructedan actor-critic-identifier architecture to deal with the infi-nite horizon optimal control of uncertain nonlinear systems,characterized by the introduction of a robust dynamic neuralnetwork.

In this paper, we employ the online policy iteration algo-rithm to tackle the decentralized control of a class of non-linear interconnected systems. To design the decentralizedcontrol scheme of the overall system, the optimal controllersof the isolated subsystems are designed at first with thecost functions modified to account for the interconnections.Then, the decentralized control strategy can be establishedby adding appropriate feedback gains to the local optimalcontrol policies. Next, the online policy iteration algorithmis developed to solve the HJB equations related to the optimalcontrol by constructing and training some critic networks. It isshown that the approximate closed-form expressions of theoptimal control policies are available. Hence, there is no needto build action networks. Additionally, the uniform ultimateboundedness (UUB) of the dynamics of the weight estimationerrors is analyzed using the Lyapunov approach. Remark-ably, considering the effectiveness of ADP and reinforcementlearning techniques in solving the nonlinear optimal controlproblem, the decentralized control approach established here isnatural and convenient. More importantly, it can be employedto stabilize a broad class of nonlinear large-scale systems.

This paper is organized as follows. In Section II, the decen-tralized control problem of the large-scale system is described.In Section III, the optimal control of isolated subsystemsis presented in the framework of HJB equations, based onwhich, the decentralized control strategy can be developed.In Section IV, the online policy iteration algorithm is intro-duced to solve the HJB equations with convergence analysis.In addition, critic networks are constructed for facilitating theimplementation of online algorithm. The UUB of the dynamicsof the weight estimation errors is proved as well. In Section V,an example is given to demonstrate the effectiveness of theestablished approach. In Section VI, concluding remarks areprovided.

II. PROBLEM STATEMENT

In this paper, we study a class of continuous-time nonlinearlarge-scale systems composed of N interconnected subsystems

Fig. 1. Structural diagram of the decentralized control problem of theinterconnected system.

described by

xi(t) = fi (xi (t)) + gi (xi (t))(ui (xi (t)) + Zi (x(t))

)

i = 1, 2, . . . , N (1)

where xi (t) ∈ Rni and ui (xi (t)) ∈ R

mi are the state and con-trol vectors of the i th subsystem, respectively. In large-scalesystem (1), x = [

x T1 x T

2 . . . x TN

]T ∈ Rn denotes the overall

state, where n = ∑Ni=1 ni . Correspondingly, x1, x2, . . . , xN

are called local states, whereas u1(x1), u2(x2), . . . , uN (xN ) arelocal controls. Note that for subsystem i , fi (xi ), gi(xi ), andgi(xi )Zi (x) represent the nonlinear internal dynamics, inputgain matrix, and interconnected term, respectively.

Let xi (0) = xi0 be the initial state of the i th subsystem,i = 1, 2, . . . , N . Additionally, we let the following assump-tions hold throughout this paper.

Assumption 1: The state vector xi = 0 is the equilibriumof the i th subsystem, i = 1, 2, . . . , N .

Assumption 2: The functions fi (·) and gi (·) are dif-ferentiable in their arguments with fi (0) = 0, wherei = 1, 2, . . . , N .

Assumption 3: The feedback control vector ui (xi) = 0when xi = 0, where i = 1, 2, . . . , N .

Let Ri ∈ Rmi ×mi , i = 1, 2, . . . , N , be symmetric positive

definite matrices. Then, we denote Zi (x) = R1/2i Z i (x), where

Zi (x) ∈ Rmi , i = 1, 2, . . . , N , are bounded as follows:

‖Zi (x)‖ ≤N∑

j=1

ρi j hi j (x j ), i = 1, 2, . . . , N. (2)

In (2), ρi j are nonnegative constants and hi j (x j ) are positivesemidefinite functions with i, j = 1, 2, . . . , N .

If we define hi (xi ) = max{h1i (xi ), h2i (xi ), . . . , hNi (xi)},i = 1, 2, . . . , N , then (2) can be formulated as

‖Zi (x)‖ ≤N∑

j=1

λi j h j (x j ), i = 1, 2, . . . , N (3)

where λi j ≥ ρi j hi j (x j )/h j (x j ), i, j = 1, 2, . . . , N , are alsononnegative constants.

When dealing with the decentralized control problem, weaim at finding N control policies u1(x1), u2(x2), . . . , uN (xN )to stabilize the large-scale system (1). It is important to notethat in the control pair (u1(x1), u2(x2), . . . , uN (xN )), ui (xi )is only a function of the corresponding local state, namely xi ,where i = 1, 2, . . . , N . The schematic diagram of thedecentralized control problem is shown in Fig. 1.


III. DECENTRALIZED CONTROLLER DESIGN VIA

OPTIMAL CONTROL SCHEME

In this section, we investigate the methodology for decen-tralized controller design. Two sections are included in thispart. In the first section, the optimal control of the isolatedsubsystems is described under the framework of HJB equa-tions, whereas in the second section, the decentralized controlstrategy can be constructed based on the optimal controlpolicies.

A. Optimal Control and the HJB Equations

Now, we consider the N isolated subsystems correspondingto (1) that are given by

xi(t) = fi (xi(t)) + gi(xi (t))ui (xi (t)), i = 1, 2, . . . , N. (4)

For the i th isolated subsystem, we further assume thatfi + giui is Lipschitz continuous on a set �i in R

ni containingthe origin, and the subsystem is controllable in the sensethat there exists a continuous control policy on �i thatasymptotically stabilizes the subsystem.

In this paper, to deal with the infinite horizon optimalcontrol problem, we have to find the control policies ui (xi),i = 1, 2, . . . , N , which minimize the local cost functions

Ji (xi0) =∫ ∞

0

{Q2

i (xi(τ )) + uTi (xi (τ ))Ri ui (xi(τ ))

}dτ

i = 1, 2, . . . , N (5)

where Qi (xi ), i = 1, 2, . . . , N , are positive definite functionssatisfying

hi (xi) ≤ Qi (xi ), i = 1, 2, . . . , N. (6)

Based on optimal control theory, here, the designed feed-back controls must not only stabilize the subsystems on �i ,i = 1, 2, . . . , N , but also guarantee that the cost functions (5)are finite. In other words, the control policies must be admis-sible. Below is the definition of admissible control.

Definition 1: Consider the isolated subsystem i , a con-trol policy μi (xi ) is defined as admissible with respectto (5) on �i , denoted by μi ∈ �i (�i ), if μi (xi) is continuouson �i , μi (0) = 0, ui (xi ) = μi (xi ) stabilizes (4) on �i , andJi (xi0) is finite for all xi0 ∈ �i .

For any set of admissible control policies μi ∈ �i (�i ),i = 1, 2, . . . , N , if the associated cost functions

Vi (xi0) =∫ ∞

0

{Q2

i (xi (τ )) + μTi (xi (τ ))Riμi (xi (τ ))

}dτ

i = 1, 2, . . . , N (7)

are continuously differentiable, then the infinitesimal versionsof (7) are the so-called nonlinear Lyapunov equations

0 = Q2i (xi ) + μT

i (xi )Riμi (xi) + (∇Vi (xi ))T ( fi (xi )

+ gi(xi )μi (xi )), i = 1, 2, . . . , N (8)

with Vi (0) = 0, i = 1, 2, . . . , N . In (8), the terms ∇Vi (xi),i = 1, 2, . . . , N , denote the partial derivatives of the localcost functions Vi (xi) with respect to local states xi , i.e.,∇Vi (xi ) = ∂Vi (xi)/∂xi , where i = 1, 2, . . . , N .

Define the Hamiltonian functions of the N isolated subsys-tems as follows:

Hi(xi , μi ,∇Vi (xi )) = Q2i (xi ) + μT

i (xi )Riμi (xi )

+ (∇Vi(xi ))T ( fi (xi ) + gi(xi )μi (xi ))

(9)

where i = 1, 2, . . . , N .The optimal cost functions of the N isolated subsystems

can be formulated as

J ∗i (xi0) = min

μi ∈�i (�i )

∫ ∞

0

{Q2

i (xi (τ )) + μTi (xi (τ ))Ri

×μi (xi (τ ))}dτ, i = 1, 2, . . . , N. (10)

In view of optimal control theory, the optimal cost functionsJ ∗

i (xi ), i = 1, 2, . . . , N , satisfy the HJB equations

0 = minμi ∈�i (�i )

Hi(xi , μi ,∇ J ∗i (xi )), i = 1, 2, . . . , N (11)

where ∇ J ∗i (xi ) = ∂ J ∗

i (xi )/∂xi , i = 1, 2, . . . , N . Assumethat the minima on the right hand side of (11) exist and areunique. Then, the optimal control policies for the N isolatedsubsystems are

u∗i (xi ) = arg min

μi ∈�i (�i )Hi(xi , μi ,∇ J ∗

i (xi))

= −1

2R−1

i gTi (xi )∇ J ∗

i (xi ), i = 1, 2, . . . , N. (12)

Substituting the optimal control policies (12) into the nonlinearLyapunov equations (8), we can obtain the formulation ofthe HJB equations in terms of ∇ J ∗

i (xi), i = 1, 2, . . . , N , asfollows:

0 = Q2i (xi ) + (∇ J ∗

i (xi ))T fi (xi )

−1

4(∇ J ∗

i (xi ))T gi (x)R−1


i (xi ) (13)

with J ∗i (0) = 0 and i = 1, 2, . . . , N .

Remark 1: The formulas developed in (12) display an arrayof closed-form expressions of the optimal control policies,which obviates the need to search for the optimal controlpolicies via optimization process. However, the knowledgeof J ∗

i (xi ), i = 1, 2, . . . , N , is required, which implies theimportance of the solutions of HJB equations.

B. Decentralized Control Strategy

According to (12), we have expressed the optimal controlpolicies, i.e., u∗

1(x1), u∗2(x2), . . . , u∗

N (xN ), for the N iso-lated subsystems (4). In the following, we will show thatby proportionally increasing some local feedback gains, astabilizing decentralized control scheme can be established forthe interconnected system (1). Now, we give the followinglemma, indicating how the feedback gains can be added, toguarantee the asymptotic stability of the isolated subsystems.

Lemma 1: Consider the isolated subsystems (4), the feed-back controls

ui (xi ) = πi u∗i (xi)

= −1

2πi R−1


i (xi ), i = 1, 2, . . . , N (14)


ensure that the N closed-loop isolated subsystems are asymp-totically stable for all πi ≥ 1/2, where i = 1, 2, . . . , N .

Proof: The lemma can be proved by showing J ∗i (xi),

i = 1, 2, . . . , N , are Lyapunov functions. First of all, inlight of (10), we can find that J ∗

i (xi ) > 0 for any xi = 0and J ∗

i (xi ) = 0 when xi = 0, which implies that J ∗i (xi),

i = 1, 2, . . . , N , are positive definite functions. Next, thederivatives of J ∗

i (xi ), i = 1, 2, . . . , N , along the correspond-ing trajectories of the closed-loop isolated subsystems aregiven by

J ∗i (xi) = (∇ J ∗

i (xi ))T xi

= (∇ J ∗i (xi ))

T ( fi (xi ) + gi(xi )ui (xi )) (15)

where i = 1, 2, . . . , N . Then, by adding and subtract-ing (1/2)(∇ J ∗

i (xi ))T gi(xi )u∗

i (xi ) to (15) and considering(12)–(14), we have

J ∗i (xi ) = (∇ J ∗

i (xi ))T fi (xi )

− 1

4(∇ J ∗

i (xi ))T gi (xi )R−1


i (xi )

− 1

2

(πi − 1

2

)(∇ J ∗

i (xi))T

× gi(xi )R−1i gT

i (xi )∇ J ∗i (xi)

= −Q2i (xi ) − 1

2

(πi − 1

2

)∥∥∥R−1/2


i (xi )∥∥∥

2

(16)

where i = 1, 2, . . . , N . Observing (16), we can obtainthat J ∗

i (xi) < 0 for all πi ≥ 1/2 and xi = 0, wherei = 1, 2, . . . , N . Therefore, the conditions for Lyapunov localstability theory are satisfied and the proof is completed.

Remark 2: Lemma 1 reveals that any feedback controlsui (xi ), i = 1, 2, . . . , N , can ensure the asymptotic stabilityof the closed-loop isolated subsystems as long as πi ≥ 1/2,i = 1, 2, . . . , N . However, only when πi = 1, i =1, 2, . . . , N , the feedback controls are optimal. In fact, similarresults have been given in [47]–[49], showing that the optimalcontrols u∗

i (xi ), i = 1, 2, . . . , N , are robust in the sense thatthey have infinite gain margins.

Now, we present the main theorem of this paper basedon that the acquired decentralized control strategy can beestablished.

Theorem 1: For interconnected system (1), there exist Npositive numbers π∗

i > 0, i = 1, 2, . . . , N , such thatfor any πi ≥ π∗

i , i = 1, 2, . . . , N , the feedback controlsdeveloped by (14) ensure that the closed-loop interconnectedsystem is asymptotically stable. In other words, the controlpair (u1(x1), u2(x2), . . . , uN (xN )) is the decentralized controlstrategy of large-scale system (1).

Proof: In accordance with Lemma 1, we observe thatJ ∗

i (xi), i = 1, 2, . . . , N , are all Lyapunov functions. Here,we select a composite Lyapunov function given by

L(x) =N∑

i=1

θi J ∗i (xi ) (17)

where θi , i = 1, 2, . . . , N , are arbitrary positive constants.

Taking the time derivative of L(x) along the trajectories ofthe closed-loop interconnected system, we can obtain

L(x) =N∑

i=1

θi J ∗i (xi )

=N∑

i=1

θi{(∇ J ∗

i (xi ))T ( fi (xi ) + gi (xi )ui (xi ))

+ (∇ J ∗i (xi ))

T gi(xi )Zi (x)}. (18)

Then, taking (3), (6), and (16) into consideration, (18) can beturned into the following form:

L(x) ≤ −N∑

i=1

θi

{Q2

i (xi)

+ 1

2

(πi − 1

2

)∥∥∥R−1/2


i (xi )∥∥∥

2

−∥∥∥(∇ J ∗

i (xi ))T gi(xi )R−1/2

i

∥∥∥‖Zi (x)‖

}

≤ −N∑

i=1

θi

{Q2

i (xi)

+ 1

2

(πi − 1

2

)∥∥∥(∇ J ∗

i (xi ))T gi (xi )R−1/2

i

∥∥∥

2

−∥∥∥(∇ J ∗

i (xi ))T gi(xi )R−1/2

i

∥∥∥

N∑

j=1

λi j Q j (x j )

}.

(19)

Here, we denote

= diag{θ1, θ2, . . . , θN } (20)

� =

⎡

⎢⎢⎢⎣

λ11 λ12 · · · λ1N

λ21 λ22 · · · λ2N...

.... . .

...λN1 λN2 · · · λN N

⎤

⎥⎥⎥⎦

(21)

and

� = diag

{1

2

(π1− 1

2

),

1

2

(π2− 1

2

), . . . ,

1

2

(πN − 1

2

)}. (22)

Therefore, through introducing a 2N-dimensional vector

ξ =

⎡

⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎣

Q1(x1)Q2(x2)

...QN (xN )

− − − − − − − − − − − − −∥∥∥(∇ J ∗

1 (x1))T g1(x1)R−1/2

1

∥∥∥

∥∥∥(∇ J ∗

2 (x2))T g2(x2)R−1/2

2

∥∥∥

...∥∥∥(∇ J ∗

N (xN ))T gN (xN )R−1/2N

∥∥∥

⎤

⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎦

(23)


we can transform (19) to the following compact form:

L(x) ≤ − ξT

⎡

⎢⎣

− 1

2�T

−1

2� �

⎤

⎥⎦ ξ

� − ξT A ξ. (24)

According to (24), sufficiently large πi , i = 1, 2, . . . , N ,can be chosen such that the matrix A is positive definite.That is to say, there exist π∗

i , i = 1, 2, . . . , N , so that anyπi ≥ π∗

i , i = 1, 2, . . . , N , are large enough to guar-antee the positive definiteness of A . Then, we haveL(x) < 0. Therefore, the closed-loop interconnected systemis asymptotically stable under the action of the control pair(u1(x1), u2(x2), . . . , uN (xN )). The proof is completed.

Clearly, the focal point of designing the decentralized con-trol strategy becomes to derive the optimal controllers for theN isolated subsystems on the basis of Theorem 1. Then, weshould put our emphasis on solving the HJB equations, whichyet, is regarded as a difficult task [12], [13]. Hence, inthe following, we will employ a more pragmatic approachto obtain the approximate solutions based on online policyiteration algorithm and neural network techniques.

IV. NEURAL-NETWORK-BASED LEARNING OPTIMAL

CONTROL OF THE ISOLATED SUBSYSTEMS USING

ONLINE POLICY ITERATION ALGORITHM

Three sections are included here. In the first section, theonline policy algorithm is introduced to tackle the optimalcontrol problem of the isolated subsystems, whereas the neuralnetwork implementation process is given in the second section.The stability proof of the dynamics of the estimation errors isdeveloped in the last section.

A. Online Policy Iteration Algorithm and Its Convergence

Here, the online policy iteration algorithm is introduced tosolve the HJB equations. The policy iteration algorithm con-sists of policy evaluation based on (8) and policy improvementbased on (12), as shown in [22]. Specifically, its iterationprocedure can be described as follows.Step 1: Choose a small positive number ε. Let p = 0 and

V (0)i (xi ) = 0, where i = 1, 2, . . . , N . Then, start

with N initial admissible control policies μ(0)1 (x1),

μ(0)2 (x2), . . . , μ

(0)N (xN ).

Step 2: Based on the control policies μ(p)i (xi ), i =

1, 2, . . . , N , solve the following nonlinear Lyapunovequations

0 = Q2i (xi ) +

(μ

(p)i (xi)

)TRiμ

(p)i (xi )

+(∇V (p+1)

i (xi ))T(

fi (xi ) + gi (xi )μ(p)i (xi)

)(25)

with V (p+1)i (0) = 0 and i = 1, 2, . . . , N .

Step 3: Update the control policies via

μ(p+1)i (xi) = −1

2R−1

i gTi (xi )∇V (p+1)

i (xi ) (26)

where i = 1, 2, . . . , N.

Step 4: If ‖V (p+1)i (xi ) − V (p)

i (xi )‖ ≤ ε, i = 1, 2, . . . , N ,stop and obtain the approximate optimal controls ofthe N isolated subsystems; else, let p = p + 1 andgo back to Step 2.

Note that N initial admissible control policies are requiredin the above algorithm. In the following, we present theconvergence analysis of the online policy iteration algorithmfor the isolated subsystems.

Theorem 2: Consider the N isolated subsystems (4), givenN initial admissible control policies μ

(0)1 (x1), μ

(0)2 (x2),

. . . , μ(0)N (xN ). Then, using the policy iteration algorithm

established in (25) and (26), the cost functions and controlpolicies converge to the optimal ones as p → ∞, i.e.,V (p)

i (xi ) → J ∗i (xi ) and μ

(p)i (xi ) → u∗

i (xi ) as p → ∞, wherei = 1, 2, . . . , N .

Proof: First, we consider the subsystem i . Accordingto [41] and [44], when given an initial admissible controlpolicy μ

(0)i (xi), we have μ

(p)i (xi ) ∈ �i (�i ) for any p ≥ 0.

Additionally, for any ζ > 0, there exists an integer p0i , suchthat for any p ≥ p0i , the formulas

supxi∈�i

∣∣V (p)

i (xi ) − J ∗i (xi )

∣∣ < ζ (27)

andsup

xi∈�i

∣∣μ(p)

i (xi ) − u∗i (xi )

∣∣ < ζ (28)

hold simultaneously.Next, we consider the N isolated subsystems. When given

μ(0)1 (x1), μ

(0)2 (x2), . . . , μ

(0)N (xN ), where μ

(0)i (xi ) is the ini-

tial admissible control policy corresponding to the i th sub-system, we can acquire that μ

(p)i (xi ) ∈ �i (�i ) for any

p ≥ 0, where i = 1, 2, . . . , N . In addition, we denotep0 = max{p01, p02, . . . , p0N }. Thus, we can conclude thatfor any ζ > 0, there exists an integer p0, such that forany p ≥ p0, (27) and (28) are true with i = 1, 2, . . . , N .In other words, the algorithm will converge to the optimalcost functions and optimal control policies of the N isolatedsubsystems. The proof is completed.

B. Implementation Procedure via Neural Networks

For the N isolated subsystems, assume that the cost func-tions Vi (xi ), i = 1, 2, . . . , N , are continuously differentiable.Then, according to the universal approximation property ofneural networks, Vi (xi ) can be reconstructed by a single-layerneural network on a compact set �i as

Vi (xi ) = ωTciσci (xi ) + εci (xi ), i = 1, 2, . . . , N (29)

where ωci ∈ Rli is the ideal weight, σci (xi ) ∈ R

li is theactivation function, li is the number of neurons in the hiddenlayer, and εci (xi ) is the approximation error of the i th neuralnetwork, i = 1, 2, . . . , N .

The derivatives of the cost functions with respect to theirstate vectors are formulated as

∇Vi (xi ) = (∇σci (xi))T ωci + ∇εci (xi ),

i = 1, 2, . . . , N (30)


where ∇σic(xi ) = ∂σci (xi )/∂xi ∈ Rli ×ni and ∇εci (xi ) =

∂εci (xi )/∂xi ∈ Rni are the gradient of the activation function

and approximation error of the i th neural network, respec-tively, i = 1, 2, . . . , N . Based on (30), the Lyapunov equa-tions (8) becomes

0 = Q2i (xi ) +μT

i Riμi +(ωT

ci∇σci (xi ) + (∇εci (xi ))T )

xi (31)

where i = 1, 2, . . . , N.For the i th neural network, i = 1, 2, . . . , N , assume that

the neural network weight vector ωci , the gradient ∇σci (xi),and the approximation error εci (xi ) and its derivative ∇εci (xi )are all bounded on the compact set �i . In addition, accordingto [43], we have εci (xi ) → 0 and ∇εci (xi ) → 0 as li → ∞,where i = 1, 2, . . . , N .

Because the ideal weights are unknown, N critic neuralnetworks can be built to approximate the cost functions asfollows:

Vi (xi ) = ωTciσci (xi ), i = 1, 2, . . . , N (32)

where ωci , i = 1, 2, . . . , N , is the estimated weights. Here,σci (xi ), i = 1, 2, . . . , N , is selected such that Vi (xi ) > 0 forany xi = 0 and Vi (xi ) = 0 when xi = 0.

Similarly, the derivatives of the approximate cost functionswith respect to the state vectors can be expressed by

∇ Vi (xi ) = (∇σci (xi))T ωci , i = 1, 2, . . . , N (33)

where ∇ Vi (xi ) = ∂ Vi (xi )/∂xi , i = 1, 2, . . . , N . Then, theapproximate Hamiltonian functions can be expressed as

Hi(xi , μi , ωci ) = Q2i (xi ) + μT

i Riμi + ωTci∇σci (xi )xi

= eci , i = 1, 2, . . . , N. (34)

For the purpose of training the critic networks of the isolatedsubsystems, it is desired to design ωci , i = 1, 2, . . . , N , tominimize the following objective functions:

Eci = 1

2eT

ci eci , i = 1, 2, . . . , N. (35)

The standard steepest descent algorithm is introduced to tunethe critic networks, then their weights are updated through

˙ωci = −αci

[∂ Eci

∂ωci

], i = 1, 2, . . . , N (36)

where αci > 0, i = 1, 2, . . . , N , is the learning rates of thecritic networks.

On the other hand, based on (30), the Hamiltonian functionstake the following forms:

Hi(xi , μi , ωci ) = Q2i (xi ) + μT

i Riμi + ωTci∇σci (xi )xi

= ecHi , i = 1, 2, . . . , N (37)

where ecHi = −(∇εci (xi ))T xi , i = 1, 2, . . . , N , is the

residual errors because of the neural network approximation.Denote δi = ∇σci (xi)xi , i = 1, 2, . . . , N . We assume that

there exist N positive constants δMi , i = 1, 2, . . . , N , suchthat

‖δi‖ ≤ δMi , i = 1, 2, . . . , N. (38)

In addition, we define the weight estimation errors of the criticnetworks as ωci = ωci − ωci , where i = 1, 2, . . . , N . Then,combining (34) with (37) yields

ecHi − eci = ωTciδi , i = 1, 2, . . . , N. (39)

Therefore, the dynamics of the weight estimation errors canbe given as follows:

˙ωci = αci (ecHi − ωTciδi )δi , i = 1, 2, . . . , N. (40)

Incidentally, the persistency of excitation condition isrequired to tune the i th critic network to guarantee that‖δi‖ ≥ δmi , where δmi , i = 1, 2, . . . , N , are positiveconstants. Thus, a set of probing noises will be added to theisolated subsystems to satisfy the condition in practice.

When implementing the online policy iteration algorithm,to accomplish the policy improvement, we should obtainthe control polices that minimize the current cost functions.Hence, according to (12) and (30), we have

μi (xi ) = −1

2R−1

i gTi (xi )∇Vi (xi)

= −1

2R−1

i gTi (xi )

((∇σci (xi ))

T ωci + ∇εci (xi ))

(41)

where i = 1, 2, . . . , N . Correspondingly, the approximatecontrol policies can be obtained by

μi (xi ) = −1

2R−1

i gTi (xi )∇ Vi (xi )

= −1

2R−1

i gTi (xi )(∇σci (xi ))

T ωci (42)

where i = 1, 2, . . . , N .Remark 3: According to (42), it is obvious to observe that

the approximate control policies of the N isolated subsystemscan be derived directly based on the trained critic networks.Therefore, unlike the traditional actor-critic architecture, theaction neural networks are not required any more.

C. Stability Analysis

When considering the critic networks, the weight estimationdynamics are UUB as described in the following theorem.

Theorem 3: For the N isolated subsystems (4), the weightupdate laws for tuning the critic networks are given by (36).Then, the dynamics of the weight estimation errors of the criticnetworks are UUB.

Proof: Choose N Lyapunov function candidates describedas follows:

Li (t) = 1

αcitr(ωT

ci ωci), i = 1, 2, . . . , N. (43)

The time derivatives of the Lyapunov functions Li (t),i = 1, 2, . . . , N , along the trajectories of the error dynam-ics (40) are

Li (t) = 2

αcitr(ωT

ci˙ωci

)

= 2

αcitr(ωT

ciαci(ecHi − ωT

ciδi)δi

)(44)


Fig. 2. Convergence of the weight vector of the critic network 1 (ωac11,ωac12, and ωac13 represent ωc11, ωc12, and ωc13, respectively).

Fig. 3. Convergence of the weight vector of the critic network 2 (ωac21,ωac22, and ωac23 represent ωc21, ωc22, and ωc23, respectively).

where i = 1, 2, . . . , N . After some basic manipulations, ityields

Li(t) ≤ −(2 − αci )∥∥ωT

ciδi∥∥2 + 1

αcie2

cHi (45)

where i = 1, 2, . . . , N . In view of the Cauchy–Schwarzinequality and (38), we can conclude that Li (t) < 0 as longas

⎧⎪⎪⎨

⎪⎪⎩

0 < αci < 2

‖ωci‖ >

√e2

cHi

αci (2 − αci )δ2Mi

(46)

where i = 1, 2, . . . , N . In accordance with the Lyapunovstability theory, we obtain that the dynamics of the weightestimation errors of the critic networks are all UUB. Mean-while, the norms of the weight estimation errors are boundedas well. The proof is completed.

Remark 4: Let ûi (xi ) = πi μi (xi ), where μi (xi),i = 1, 2, . . . , N , are obtained by (42). According to theselections of the activation functions of the critic networks,

Fig. 4. 3-D plot of the approximation error of the cost function of isolatedsubsystem 1, i.e., J∗

1 (x1) − V1(x1).

Fig. 5. 3-D plot of the approximation error of the control policy of isolatedsubsystem 1, i.e., u∗

1(x1) − μ1(x1).

we can easily find that the approximate optimal cost func-tions Vi (xi ), i = 1, 2, . . . , N , are also Lyapunov functions.Furthermore, similar to the proof of Theorem 1, we haveL(x) ≤ − ξT A ξ + �e, where �e is the sum of the approx-imation errors. Hence, we can conclude that based on theapproximate optimal control policies μi (xi ), i = 1, 2, . . . , N ,the developed control pair ( û1(x1), û2(x2), . . . , ûN (xN )) canensure the UUB of the state trajectories of the closed-loopinterconnected system. It is in this sense that we accomplishthe design of the decentralized control scheme by adoptingthe learning optimal control approach based on online policyiteration algorithm.

Remark 5: Note that the controller presented here is adecentralized stabilization one. Though the optimal decentral-ized controller of interconnected systems has been studiedbefore [50], in this paper, we aim at developing a noveldecentralized control strategy based on ADP. How to extendthe present results to the design of optimal decentralizedcontrol for nonlinear interconnected systems is part of ourfuture research.


Fig. 6. 3-D plot of the approximation error of the cost function of isolatedsubsystem 2, i.e., J∗

2 (x2) − V2(x2).

V. SIMULATION STUDY

A simulation example is provided in this section to show theapplicability of the decentralized control strategy establishedin this paper.

Consider the following continuous-time nonlinear large-scale system consisting of two interconnected subsystems:

x1 =[ −x11 + x12

−0.5x11 − 0.5x12 − 0.5x12(cos(2x11) + 2)2

]

+[

0cos(2x11) + 2

](u1(x1)+(x11+x22) sin x2

12 cos(0.5x21))

x2 =[

x22

−x21 − 0.5x22 + 0.5x221x22

]

+[

0x21

] (u2(x2) + 0.5(x12 + x22) cos

(ex2

21))

(47)

where x1 = [x11 x12]T ∈ R2 and u1(x1) ∈ R are the state and

control variables of subsystem 1, and x2 = [x21 x22]T ∈ R2

and u2(x2) ∈ R are the state and control variables of subsys-tem 2. Let R1 = R2 = I , where I denotes the identity matrixwith suitable dimension. Additionally, let h1(x1) = ‖x1‖ andh2(x2) = |x22|. Then, we find that Z1(x) and Z2(x) withx = [x T

1 x T2 ]T are upper bounded as in (3). For example, we

can select λ11 = λ12 = 1 and λ21 = λ22 = 1/2.To design the decentralized controller of interconnected

system (47), we first deal with the optimal control problemof two isolated subsystems. Here, we choose Q1(x1) = ‖x1‖and Q2(x2) = |x22|. Hence, the cost functions of the optimalcontrol problem are

J1(x10) =∫ ∞

0

{x2

11 + x212 + uT

1 u1}dτ (48)

and

J2(x20) =∫ ∞

0

{x2

22 + uT2 u2

}dτ. (49)

We adopt the online policy iteration algorithm totackle the optimal control problem, where two critic net-works are constructed to approximate the cost functions.We denote the weight vectors of the two critic networks as

Fig. 7. 3-D plot of the approximation error of the control policy of isolatedsubsystem 2, i.e., u∗

2(x2) − μ2(x2).

Fig. 8. State trajectory of subsystem 1 under the action of the decentralizedcontrol strategy (π1μ1(x1), π2μ2(x2)).

ωc1 = [ωc11 ωc12 ωc13]T and ωc2 = [ωc21 ωc22 ωc23]T .During the simulation process, the initial weights of the criticnetworks are chosen randomly in [0, 2]. In addition, theactivation functions of the two critic networks are chosen asσc1(x1) = [x2

11 x11x12 x212]T and σc2(x2) = [x2

21 x21x22 x222]T .

Besides, let the learning rates of the critic networks beαc1 = αc2 = 0.1 and the initial states of the two isolatedsubsystems be x10 = x20 = [1 − 1]T .

During the implementation process of the online policyiteration algorithm, for each isolated subsystem, we add aprobing noise to satisfy the persistency of excitation condition.We can observe that the convergence results of the weightshave occurred after 750 and 180 s, respectively. Then, theprobing signals are turned off. Actually, the weights of thecritic networks converge to

ωc1 = [0.498969 0.000381 0.999843]T (50)

and

ωc2 = [1.000002 − 0.000021 0.999992]T (51)

that are shown in Figs. 2 and 3.


Fig. 9. State trajectory of subsystem 2 under the action of the decentralizedcontrol strategy (π1μ1(x1), π2μ2(x2)).

Based on the convergent weights ωc1 and ωc2, we canobtain the approximate optimal cost function and controlpolicy for each isolated subsystem, namely, V1(x1), μ1(x1),V2(x2), and μ2(x2). In comparison, for the method proposedin [43], the optimal cost function and control policy of isolatedsubsystem 1 are J ∗

1 (x1) = 0.5x211 + x2

12 and u∗1(x1) =

−(cos(2x11) + 2)x12, respectively. Similarly, the optimal costfunction and control policy of isolated subsystem 2 areJ ∗

2 (x2) = x221 + x2

22 and u∗2(x2) = −x21x22.

Therefore, for isolated subsystem 1, the error between theoptimal cost function and the approximate one is shown inFig. 4. In addition, the error between the optimal control policyand the approximate version is shown in Fig. 5. It is clear tosee that both the approximation errors are close to zero, whichverifies the good performance of the online learning algorithm.When regarding the isolated subsystem 2, we obtain the samesimulation results shown in Figs. 6 and 7.

Next, by choosing θ1 = θ2 = 1 and π1 = π2 = 2, wecan guarantee the positive definiteness of the matrix A . Thus,(π1μ1(x1), π2μ2(x2)) is the decentralized control strategy ofthe original interconnected system (47). Here, we apply thedecentralized control scheme to controlled plant (47) for 40 sand obtain the evolution processes of the state trajectoriesshown in Figs. 8 and 9. Through zooming in the statetrajectories near the zero, it is demonstrated that the state tra-jectories of the closed-loop system are UUB. Obviously, thesesimulation results authenticate the validity of the decentralizedcontrol approach developed in this paper.

VI. CONCLUSION

In this paper, a novel decentralized control strategy isdeveloped to deal with the stabilization problem of a classof continuous-time nonlinear large-scale systems using onlinepolicy iteration algorithm. Initially, the optimal controllersof the isolated subsystems are designed. Then, it is shownthat the decentralized control strategy of the overall systemcan be established by adding feedback gains to the obtainedoptimal control policies. In addition, the online policy iterationalgorithm is introduced to solve the HJB equations iteratively.

The cost functions can be approximated by constructing sev-eral critic networks and the expressions of the control policiescan be obtained directly. In addition, the dynamics of theestimation errors are proved to be UUB. Simulation studyis presented to demonstrate the validity of the decentralizedcontrol strategy in the end.

REFERENCES

[1] L. Bakule, “Decentralized control: An overview,” Annu. Rev. Control,vol. 32, no. 1, pp. 87–98, Apr. 2008.

[2] D. D. Siljak and A. I. Zecevic, “Control of large-scale systems: Beyonddecentralized feedback,” Annu. Rev. Control, vol. 29, no. 2, pp. 169–179,Dec. 2005.

[3] J. Lavaei, “Decentralized implementation of centralized controllers forinterconnected systems,” IEEE Trans. Autom. Control, vol. 57, no. 7,pp. 1860–1865, Jul. 2012.

[4] H. F. Grip, A. Saberi, and T. A. Johansen, “Observers for interconnectednonlinear and linear systems,” Automatica, vol. 48, no. 7, pp. 1339–1346,Jul. 2012.

[5] S. Mehraeen, S. Jagannathan, and M. L. Crow, “Power system stabi-lization using adaptive neural network-based dynamic surface control,”IEEE Trans. Power Syst., vol. 26, no. 2, pp. 669–680, May 2011.

[6] K. Kalsi, J. Lian, and S. H. Zak, “Decentralized dynamic outputfeedback control of nonlinear interconnected systems,” IEEE Trans.Autom. Control, vol. 55, no. 8, pp. 1964–1970, Aug. 2010.

[7] Z. G. Hou, M. M. Gupta, P. N. Nikiforuk, M. Tan, and L. Cheng,“A recurrent neural network for hierarchical control of intercon-nected dynamic systems,” IEEE Trans. Neural Netw., vol. 18, no. 2,pp. 466–481, Mar. 2007.

[8] A. Saberi, “On optimality of decentralized control for a classof nonlinear interconnected systems,” Automatica, vol. 24, no. 1,pp. 101–104, Jan. 1988.

[9] P. J. Werbos, “Advanced forecasting methods for global crisis warn-ing and models of intelligence,” in Proc. General Syst., Jun. 1977,pp. 25–38.

[10] P. J. Werbos, “Approximate dynamic programming for real-time controland neural modeling,” in Handbook of Intelligent Control: Neural,Fuzzy, and Adaptive Approaches, D. A. White and D. A. Sofge, Eds.New York, NY, USA: Van Nostrand Reinhold, 1992, ch. 13.

[11] H. Zhang, D. Liu, Y. Luo, and D. Wang, Adaptive Dynamic Pro-gramming for Control: Algorithms and Stability. London, U.K.:Springer-Verlag, 2013.

[12] F. Y. Wang, H. Zhang, and D. Liu, “Adaptive dynamic programming:An introduction,” IEEE Comput. Intell. Mag., vol. 4, no. 2, pp. 39–47,May 2009.

[13] F. L. Lewis and D. Vrabie, “Reinforcement learning and adaptivedynamic programming for feedback control,” IEEE Circuits Syst. Mag.,vol. 9, no. 3, pp. 32–50, Jul. 2009.

[14] J. Fu, H. He, and X. Zhou, “Adaptive learning and control for MIMOsystem based on adaptive dynamic programming,” IEEE Trans. NeuralNetw., vol. 22, no. 7, pp. 1133–1148, Jul. 2011.

[15] F. Y. Wang, N. Jin, D. Liu, and Q. Wei, “Adaptive dynamic programmingfor finite-horizon optimal control of discrete-time nonlinear systems withε-error bound,” IEEE Trans. Neural Netw., vol. 22, no. 1, pp. 24–36,Jan. 2011.

[16] F. L. Lewis and D. Liu, Reinforcement Learning and ApproximateDynamic Programming for Feedback Control. New York, NY, USA:Wiley, 2013.

[17] S. N. Balakrishnan, J. Ding, and F. L. Lewis, “Issues on stability ofADP feedback controllers for dynamic systems,” IEEE Trans. Syst.,Man, Cybern. B, Cybern., vol. 38, no. 4, pp. 913–917, Aug. 2008.

[18] A. Al-Tamimi, F. L. Lewis, and M. Abu-Khalaf, “Discrete-time non-linear HJB solution using approximate dynamic programming: Conver-gence proof,” IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 38, no.4, pp. 943–949, Aug. 2008.

[19] D. P. Bertsekas, M. L. Homer, D. A. Logan, S. D. Patek, andN. R. Sandell, “Missile defense and interceptor allocation by neuro-dynamic programming,” IEEE Trans. Syst., Man, Cybern. A, Syst.Humans, vol. 30, no. 1, pp. 42–51, Jan. 2000.

[20] J. Si and Y. T. Wang, “On-line learning control by association andreinforcement,” IEEE Trans. Neural Netw., vol. 12, no. 2, pp. 264–276,Mar. 2001.

[21] D. V. Prokhorov and D. C. Wunsch, “Adaptive critic designs,”IEEE Trans. Neural Netw., vol. 8, no. 5, pp. 997–1007, Sep. 1997.


[22] R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction.Cambridge, MA, USA: MIT Press, 1998.

[23] H. Zhang, Y. Luo, and D. Liu, “Neural-network-based near-optimalcontrol for a class of discrete-time affine nonlinear systems with controlconstraints,” IEEE Trans. Neural Netw., vol. 20, no. 9, pp. 1490–1503,Sep. 2009.

[24] H. Zhang, Q. Wei, and D. Liu, “An iterative adaptive dynamic pro-gramming method for solving a class of nonlinear zero-sum differentialgames,” Automatica, vol. 47, no. 1, pp. 207–214, Jan. 2011.

[25] D. Wang, D. Liu, and Q. Wei, “Finite-horizon neuro-optimal trackingcontrol for a class of discrete-time nonlinear systems using adaptivedynamic programming approach,” Neurocomputing, vol. 78, no. 1,pp. 14–22, Feb. 2012.

[26] D. Liu, D. Wang, D. Zhao, Q. Wei, and N. Jin, “Neural-network-basedoptimal control for a class of unknown discrete-time nonlinear systemsusing globalized dual heuristic programming,” IEEE Trans. Autom. Sci.Eng., vol. 9, no. 3, pp. 628–634, Jul. 2012.

[27] D. Liu, H. Li, and D. Wang, “H∞ control of unknown discrete-time nonlinear systems with control constraints using adaptive dynamicprogramming,” in Proc. Int. Joint Conf. Neural Netw., Jun. 2012,pp. 3056–3061.

[28] D. Liu, D. Wang, and X. Yang, “An iterative adaptive dynamic program-ming algorithm for optimal control of unknown discrete-time nonlinearsystems with constrained inputs,” Inf. Sci., vol. 220, no. 20, pp. 331–342,Jan. 2013.

[29] T. Dierks and S. Jagannathan, “Online optimal control of affine nonlineardiscrete-time systems with unknown internal dynamics by using time-based policy update,” IEEE Trans. Neural Netw. Learn. Syst., vol. 23,no. 7, pp. 1118–1129, Jul. 2012.

[30] Y. Jiang and Z. P. Jiang, “Robust adaptive dynamic programming forlarge-scale systems with an application to multimachine power systems,”IEEE Trans. Circuits Syst. II, Exp. Briefs, vol. 59, no. 10, pp. 693–697,Oct. 2012.

[31] H. Xu, S. Jagannathan, and F. L. Lewis, “Stochastic optimal control ofunknown linear networked control system in the presence of randomdelays and packet losses,” Automatica, vol. 48, no. 6, pp. 1017–1030,Jun. 2012.

[32] S. Mehraeen and S. Jagannathan, “Decentralized optimal control of aclass of interconnected nonlinear discrete-time systems by using onlineHamilton-Jacobi-Bellman formulation,” IEEE Trans. Neural Netw.,vol. 22, no. 11, pp. 1757–1769, Nov. 2011.

[33] J. W. Park, R. G. Harley, and G. K. Venayagamoorthy, “Decentralizedoptimal neuro-controllers for generation and transmission devices inan electric power network,” Eng. Appl. Artif. Intell., vol. 18, no. 1,pp. 37–46, Feb. 2005.

[34] J. Liang, G. K. Venayagamoorthy, and R. G. Harley, “Wide-area mea-surement based dynamic stochastic optimal power flow control for smartgrids with high variability and uncertainty,” IEEE Trans. Smart Grid,vol. 3, no. 1, pp. 59–69, Mar. 2012.

[35] H. N. Wu and B. Luo, “Neural network based online simultaneous policyupdate algorithm for solving the HJI equation in nonlinear H∞ control,”IEEE Trans. Neural Netw. Learn. Syst., vol. 23, no. 12, pp. 1884–1895,Dec. 2012.

[36] S. G. Khan, G. Herrmann, F. L. Lewis, T. Pipe, and C. Melhuish,“Reinforcement learning and optimal adaptive control: An overview andimplementation examples,” Annu. Rev. Contorl, vol. 36, no. 1, pp. 42–59,Apr. 2012.

[37] Z. Ni, H. He, and J. Wen, “Adaptive learning in tracking control basedon the dual critic network design,” IEEE Trans. Neural Netw. Learn.Syst., vol. 24, no. 6, pp. 913–928, Jun. 2013.

[38] X. Xu, Z. Hou, C. Lian, and H. He, “Online learning control usingadaptive critic designs with sparse kernel machines,” IEEE Trans. NeuralNetw. Learn. Syst., vol. 24, no. 5, pp. 762–775, May 2013.

[39] Y. Jiang and Z. P. Jiang, “Computational adaptive optimal control forcontinuous-time linear systems with completely unknown dynamics,”Automatica, vol. 48, no. 10, pp. 2699–2704, Oct. 2012.

[40] A. Heydari and S. N. Balakrishnan, “Finite-horizon control-constrainednonlinear optimal control using single network adaptive critics,” IEEETrans. Neural Netw. Learn. Syst., vol. 24, no. 1, pp. 145–157, Jan. 2013.

[41] M. Abu-Khalaf and F. L. Lewis, “Nearly optimal control laws fornonlinear systems with saturating actuators using a neural network HJBapproach,” Automatica, vol. 41, no. 5, pp. 779–791, May 2005.

[42] D. Vrabie and F. L. Lewis, “Neural network approach to continuous-time direct adaptive optimal control for partially unknown nonlinearsystems,” Neural Netw., vol. 22, no. 3, pp. 237–246, Apr. 2009.

[43] K. G. Vamvoudakis and F. L. Lewis, “Online actor-critic algorithm tosolve the continuous-time infinite horizon optimal control problem,”Automatica, vol. 46, no. 5, pp. 878–888, May 2010.

[44] D. Liu, X. Yang, and H. Li, “Adaptive optimal control for a class ofcontinuous-time affine nonlinear systems with unknown internal dynam-ics,” in Neural Computing and Applications. New York, NY, USA:Springer-Verlag, Nov. 2012.

[45] S. Bhasin, M. Johnson, and W. E. Dixon, “A model-free robust policyiteration algorithm for optimal control of nonlinear systems,” in Proc.49th IEEE Conf. Decision Control, Dec. 2010, pp. 3060–3065.

[46] S. Bhasin, R. Kamalapurkar, M. Johnson, K. G. Vamvoudakis,F. L. Lewis, and W. E. Dixon, “A novel actor–critic–identifier architec-ture for approximate optimal control of uncertain nonlinear systems,”Automatica, vol. 49, no. 1, pp. 82–92, Jan. 2013.

[47] S. T. Glad, “On the gain margin of nonlinear and optimal regulators,”IEEE Trans. Autom. Control, vol. 29, no. 7, pp. 615–620, Jul. 1984.

[48] J. N. Tsitsiklis and M. Athans, “Guaranteed robustness propertiesof multivariable nonlinear stochastic optimal regulators,” IEEE Trans.Autom. Control, vol. 29, no. 8, pp. 690–696, Aug. 1984.

[49] R. W. Beard, G. N. Saridis, and J. T. Wen, “Galerkin approximationsof the generalized Hamilton-Jacobi-Bellman equation,” Automatica,vol. 33, no. 12, pp. 2159–2177, Dec. 1997.

[50] D. D. Siljak, Decentralized Control of Complex Systems.Boston, MA, USA: Academic, 2012.

Derong Liu (S’91–M’94–SM’96–F’05) receivedthe B.S. degree in mechanical engineering from theEast China Institute of Technology (now NanjingUniversity of Science and Technology), Nanjing,China, in 1982, the M.S. degree in automaticcontrol theory and applications from the Instituteof Automation, Chinese Academy of Sciences,Beijing, China, in 1987, and the Ph.D. degree inelectrical engineering from the University of NotreDame, Notre Dame, IN, USA, in 1994.

He was a Product Design Engineer with ChinaNorth Industries Corporation, Jilin, China, from 1982 to 1984. He was anInstructor with the Graduate School of the Chinese Academy of Sciences,Beijing, from 1987 to 1990. He was a Staff Fellow with the GeneralMotors Research and Development Center, Warren, MI, USA, from 1993 to1995. He was an Assistant Professor with the Department of Electrical andComputer Engineering, Stevens Institute of Technology, Hoboken, NJ, USA,from 1995 to 1999. He joined the University of Illinois at Chicago, Chicago,IL, USA, in 1999, and became a Full Professor of electrical and computerengineering and computer science in 2006. He was selected for the “100Talents Program” by the Chinese Academy of Sciences in 2008. He haspublished 14 books (six research monographs and eight edited volumes).

Dr. Liu was an Associate Editor of Automatica from 2006 to 2009. Heserves as an Associate Editor of Neurocomputing, International Journalof Neural Systems, Soft Computing, Neural Computing and Applications,Journal of Control Science and Engineering, and Science in China Series F:Information Sciences. He was an elected member of the Board of Governorsof the International Neural Network Society from 2010 to 2012. He is aGoverning Board Member of Asia Pacific Neural Network Assembly. Hewas a member of the Conference Editorial Board of the IEEE ControlSystems Society from 1995 to 2000, an Associate Editor of the IEEETRANSACTIONS ON CIRCUITS AND SYSTEMS-I: FUNDAMENTAL THEORY

AND APPLICATIONS from 1997 to 1999, the IEEE TRANSACTIONS ONSIGNAL PROCESSING from 2001 to 2003, the IEEE TRANSACTIONS

ON NEURAL NETWORKS from 2004 to 2009, the IEEE ComputationalIntelligence Magazine from 2006 to 2009, and the IEEE Circuits andSystems Magazine from 2008 to 2009, and the Letters Editor of the IEEETRANSACTIONS ON NEURAL NETWORKS from 2006 to 2008. He was theFounding Editor of the IEEE Computational Intelligence Society’s ElectronicLetter from 2004 to 2009. Currently, he is the Editor-in-Chief of the IEEETRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS

and an Associate Editor of the IEEE TRANSACTIONS ON CONTROL

SYSTEMS TECHNOLOGY and the IEEE TRANSACTIONS ON INTELLIGENT

TRANSPORTATION SYSTEMS. He is the General Chair of the 2014 IEEEWorld Congress on Computational Intelligence, Beijing, China. He was anelected AdCom member of the IEEE Computational Intelligence Societyfrom 2006 to 2008. He is the Chair of IEEE CIS Beijing Chapter. He receivedthe Michael J. Birck Fellowship from the University of Notre Dame in 1990,the Harvey N. Davis Distinguished Teaching Award from Stevens Instituteof Technology in 1997, the Faculty Early Career Development (CAREER)Award from the National Science Foundation in 1999, the University ScholarAward from the University of Illinois in 2006, and the Overseas OutstandingYoung Scholar Award from the National Natural Science Foundation ofChina in 2008. He is a member of Eta Kappa Nu and a fellow of the INNS.


Ding Wang received the B.S. degree in mathematicsfrom the Zhengzhou University of Light Industry,Zhengzhou, China, the M.S. degree in operationsresearch and cybernetics from Northeastern Uni-versity, Shenyang, China, and the Ph.D. degree incontrol theory and control engineering from theInstitute of Automation, Chinese Academy of Sci-ences, Beijing, China, in 2007, 2009, and 2012,respectively.

He is currently an Assistant Professor with TheState Key Laboratory of Management and Control

for Complex Systems, Institute of Automation, Chinese Academy of Sciences.His current research interests include adaptive dynamic programming, neuralnetworks and learning systems, and complex systems and intelligent control.

Hongliang Li received the B.S. degree in mechan-ical engineering and automation from the BeijingUniversity of Posts and Telecommunications, Bei-jing, China, in 2010. He is currently pursuing thePh.D. degree with The State Key Laboratory ofManagement and Control for Complex Systems,Institute of Automation, Chinese Academy of Sci-ences, Beijing.

He is with the University of Chinese Academyof Sciences, Beijing. His current research interestsinclude machine learning, neural networks, rein-

forcement learning, adaptive dynamic programming, and game theory.

Documents

418 IEEE TRANSACTIONS ON NEURAL NETWORKS AND …