Asynchronous Stochastic Approximation Algorithms for Networked Systems: Regime-Switching Topologies and Multiscale Structure

Copyright © by SIAM. Unauthorized reproduction of this article is prohibited.

MULTISCALE MODEL. SIMUL. c© 2013 Society for Industrial and Applied MathematicsVol. 11, No. 3, pp. 813–839

ASYNCHRONOUS STOCHASTIC APPROXIMATION ALGORITHMSFOR NETWORKED SYSTEMS: REGIME-SWITCHING

TOPOLOGIES AND MULTISCALE STRUCTURE∗

G. YIN† , QUAN YUAN† , AND LE YI WANG‡

Abstract. This work develops asynchronous stochastic approximation (SA) algorithms fornetworked systems with multiagents and regime-switching topologies to achieve consensus control.There are several distinct features of the algorithms: (1) In contrast to most of the existing consensusalgorithms, the participating agents compute and communicate in an asynchronous fashion withoutusing a global clock. (2) The agents compute and communicate at random times. (3) The regime-switching process is modeled as a discrete-time Markov chain with a finite state space. (4) Thefunctions involved are allowed to vary with respect to time; hence, nonstationarity can be handled.(5) Multiscale formulation enriches the applicability of the algorithms. In the setup, the switchingprocess contains a rate parameter ε > 0 in the transition probability matrix that characterizeshow frequently the topology switches. The algorithm uses a step-size μ that defines how fast thenetwork states are updated. Depending on their relative values, three distinct scenarios emerge.Under suitable conditions, it is shown that a continuous-time interpolation of the iterates convergesweakly either to a system of randomly switching ordinary differential equations modulated by acontinuous-time Markov chain or to a system of differential equations (an average with respect tocertain measure). In addition, a scaled sequence of tracking errors converges to either a switchingdiffusion or a diffusion. Simulation results are presented to demonstrate these findings.

Key words. asynchronous algorithm, stochastic approximation, switching model, random com-putation time, nonstationarity, consensus

AMS subject classifications. 60F17, 62L20, 65Y05, 93E10, 93E25

DOI. 10.1137/120871614

1. Introduction. This paper develops consensus algorithms under the asynchro-nous communication and random computation environments using random switchingtopologies. Consensus problems are related to many control applications that involvecoordination of multiple entities with only limited neighborhood information to reacha global goal for the entire team. Since the mid 1990s, there have been increasing andresurgent efforts devoted to the study of consensus controls of multiagent systems.The goal is to achieve a common theme such as position, speed, load distribution,etc. for the mobile agents. In [28], a discrete-time model of autonomous agents wasproposed, which can be viewed as points or particles moving in the plane with thesame speed but with different headings. Each agent updates its heading using a localrule based on the average headings of its own and its neighbors. This is in fact aspecial version of a model introduced in [24] for simulating animation of flocking andschooling behaviors. Technically, the problems considered are related to the parallelcomputation model considered in [26], which was substantially generalized in [12];see also related works in [1, 4, 6, 16, 17, 18, 19, 22, 35]. During the past decades, ahost of researchers have devoted their efforts to the study of the consensus problems;

∗Received by the editors March 28, 2012; accepted for publication (in revised form) May 29,2013; published electronically August 15, 2013. This research was supported in part by the NationalScience Foundation under DMS-1207667.

http://www.siam.org/journals/mms/11-3/87161.html†Department of Mathematics, Wayne State University, Detroit, MI 48202 ([email protected],

[email protected]).‡Department of Electrical and Computer Engineering, Wayne State University, Detroit, MI 48202

([email protected]).

813

Dow

nloa

ded

11/2

8/14

to 1

29.1

20.2

42.6

1. R

edis

trib

utio

n su

bjec

t to

SIA

M li

cens

e or

cop

yrig

ht; s

ee h

ttp://

ww

w.s

iam

.org

/jour

nals

/ojs

a.ph

p


814 G. YIN, QUAN YUAN, AND LE YI WANG

see [7, 8, 9, 10, 15, 20, 21, 23, 25, 29] and the many references therein. Many re-sults obtained thus far are for simple dynamic systems with fixed or highly simplifiedtime-varying topologies, whereas [10], [33], and [34] dealt with time-varying topologiesunder Markovian switching.

In practical implementations of consensus or coordinated control schemes, controlactions are almost always done asynchronously, especially over a large network of sub-systems. For instance, subsystems operate independently with different clocks untilthey communicate with their neighbors; communication channels operate accordingto priorities and hence transmit data at different paces and at different times; evenfor data packets transmitted at the same time from a node system, their pathwaysthrough different routes and hubs introduce different latencies and hence arrive atdifferent times. This is especially true in mobile agents when obstacles from terrainscreate interruptions, packet losses, and delays so that consensus must be performedon delayed information, which is an asynchronous operation.

In this paper, our problems are formulated to capture two aspects of the asynchro-nism: (i) Asynchronous execution of state updates at the subsystems: Each subsystemhas a randomized timer representing the internal processing time. A subsystem canupdate its state only when the timer ticks. After the state update, the timer is re-newed and internal processing resumes until the next ticking time. (ii) Asynchronousneighborhood information exchange: When a subsystem’s timer ticks, the subsys-tem will observe the states of its neighboring subsystems at that time and adjust itsown state accordingly. Since the neighboring subsystems update their states indepen-dently, the received state information will always be a delayed information, creatinganother layer of asynchronism. These concepts are illustrated in Figure 1.

11

12

n

subsystem 1 updating time

21

22

n

subsystem 2 updating time

1r

2r

n

23

S1

Sr

S2

Neighboring state Estimation by S1

Neighboring state Estimation by S2

11Y

12Y

Fig. 1. Asynchronous operations and communications of networked subsystems.Dow

nloa

ded

11/2

8/14

to 1

29.1

20.2

42.6

1. R

edis

trib

utio

n su

bjec

t to

SIA

M li

cens

e or

cop

yrig

ht; s

ee h

ttp://

ww

w.s

iam

.org

/jour

nals

/ojs

a.ph

p


ASYNCHRONOUS SA ALGORITHMS FOR NETWORKED SYSTEMS 815

This asynchronous framework introduces fundamental challenges to constrainedconsensus control problems. In the field of consensus control, most works are onunconstrained consensus; namely, as long as the states of the subsystems achieve con-sensus there are no other constraints to be satisfied. However, practical systems oftenimpose constraints on the states. For example, for power grids, all power producedwill have to be equal to the total load at steady state, even though transient powerimbalance is allowed due to storage capabilities on generators and/or capacitance andinductance on transmission lines. In a team formation for area surveillance, a team ofmobile sensors needs to be confined to the region to be covered. In parallel or cloudcomputing, steady-state service demands and service capacity must be equal. In ourprevious work [33, 34], this constraint is satisfied by employing a “link control” strat-egy in which a reduction on a state value is always balanced by an equal amount ofincrease in its neighboring subsystems.1 Asynchronous operation renders such a con-trol scheme impractical. Further complication stems from the time-varying nature ofnetwork topologies, which changes a subsystem’s neighbors randomly. Consequently,the interaction between the stochastic processes of subsystem timers and the govern-ing Markov chain for the network topologies must be carefully studied. In this work,we employ a new control strategy in which the state constraint can be asymptoticallysatisfied even though the asynchronous operation leaves the constraint unmet duringthe transient period.

Dealing with large interconnected systems such as in a communication networkwith multiple servers, it is natural to consider the distributed, asynchronous stochas-tic approximation (SA) algorithms. If synchronous SA algorithms are used, a newiteration will not begin until the current iteration is finished in all subsystems. Sincethe dimension of a networked system can be very large, the waiting time on subsys-tems will cause serious time delays. In [26], an asynchronous algorithm was proposedwhere separate processors iterated on the same system vector (with possibly differentnoise processes and/or different dynamics) and shared information in an asynchro-nous way. In [12, 13, 40], SA algorithms for parallel and distributed processing werefurther developed. The main efforts were on the study of convergence and rates ofconvergence of such algorithms.

In this paper, we concentrate on consensus-type algorithms. Here, each compo-nent in a system (with a large number of mobile agents) can be handled by differentagents, and the information can be shared by agents. To each single agent, the agentcan start the next iteration using the newest information of iteration on other com-ponents without waiting for other agents to finish. So for each component, the timeof each iteration and the number of iterations up to that moment are random. Wenote that the underlying problems introduce some new challenges, and our solutionscarry a number of new features. When representing the algorithms as discrete-timedynamic systems, the dynamics of the systems switch randomly among a finite num-ber of regimes and at random times. The modulating force of the switching processis modeled as a discrete-time Markov chain with a finite state space. In our setup,the transition probability matrix of the Markov chain includes a small parameter ε.

1The constrained consensus control problems are motivated by load sharing and resource alloca-tion problems. When node i estimates the state of node j and decides to shift a resource of amountuij to node j, this will be a reduction on node i and an increase of the equal amount to node j. Inthis sense, both node i and node j are controlled. But the decision resides with node i, and nodej receives it passively. In synchronized operation, this will guarantee that the sum of all states isa constant at all time. In contrast, in asynchronous modes, state updates occur at different times,and, as a result, the sum of the states may not be a constant during the transient period.

Dow

nloa

ded

11/2

8/14

to 1

29.1

20.2

42.6

1. R

edis

trib

utio

n su

bjec

t to

SIA

M li

cens

e or

cop

yrig

ht; s

ee h

ttp://

ww

w.s

iam

.org

/jour

nals

/ojs

a.ph

p



Henceforth, this parameter will be called the transition frequency parameter sinceit represents how frequently the state transition will take place. The SA algorithmdefines its updating speed by another small parameter μ, which will be called theadaptation step-size. The interplay of the two parameters introduces a multiscalesystem dynamics. It turns out that the difference between the parameters (ε = O(μ),ε� μ, and μ� ε) gives rise to qualitatively different behaviors with stark contrasts.

To summarize, there are several novel features of the algorithms proposed in thispaper: (1) In contrast to most of the existing consensus algorithms, the participatingagents compute and communicate in an asynchronous fashion. (2) Based on theirlocal clocks, the agents compute and communicate at random times without using aglobal clock. (3) The regime-switching process is modeled as a discrete-time Markovchain with a finite state space. (4) The functions involved are allowed to vary withrespect to time; hence, nonstationarity can be handled. (5) Multiscale formulationenriches the applicability of the algorithms.

The rest of the paper is arranged as follows. Section 2 begins with the formu-lation of a typical consensus control problem for networked systems under randomlyswitching topologies. It serves to demonstrate how this problem naturally leads toasynchronous SA algorithms under switching dynamics. Mathematical formulation ofthe problem is then presented accordingly. Section 3 focuses on the case ε = O(μ)to introduce new techniques in establishing asymptotic behavior of the algorithms.Using weak convergence methods, convergence of the algorithm is obtained. The limitbehavior of the scaled estimation errors is also analyzed. Section 4 extends the maintechniques of section 3 to the cases of ε� μ and μ� ε. It is shown that, dependingon relative scales between the transition frequency and adaptation step-size, the asyn-chronous SA algorithms demonstrate fundamentally different asymptotic behaviors.Section 5 illustrates the main findings of this paper by simulation examples. Section6 provides further remarks and discusses some open issues.

2. Formulation. Throughout this paper, | · | denotes a Euclidean norm. A pointx in a Euclidean space is a column vector; the ith component of x is denoted by xi,and 1l denotes the column vector with all components being 1. The symbol ′ denotestranspose. The notation O(y) denotes a function of y satisfying supy |O(y)|/|y| <∞. Likewise, o(y) denotes a function of y satisfying |o(y)|/|y| → 0, as y → 0. Inparticular, O(1) denotes the boundedness and o(1) indicates convergence to 0. Tofacilitate the reading, we have placed some basic formulation for consensus controlalgorithms in an appendix. Our formulation extends beyond the traditional setup. Inlieu of the simple formulation in the appendix, we allow certain nonadditive noisesto be added. More importantly, our main effort is on asynchronous computationand communication schemes. In lieu of the constraint 1l′xn = ηr at each step, weonly require such an equality to hold asymptotically. This generalizes the setup inthe appendix. Suppose that the network topology is represented by a graph G. Incontrast to the standard setting, the graph depends on a discrete-time Markov chainso it is given by G = G(αn). In our setup, the graph can take m0 possible values.The Markov chain is used to model, for example, capacity of the network, randomenvironment, and other random factors such as interrupts, rerouting of communicationchannels, etc. Thus G(αn) =

∑m0

ι=1 G(ι)I{αn=ι}. To illustrate, suppose that initiallythe Markov chain is at α0 = ι. Then the graph takes the value G(ι). At a randominstance ρ1, the first jump of the Markov chain takes place so that αρ1 = � �= ι. Thenthe graph switches to G(�) and holds that value for a random duration until the nextjump of the Markov chain takes place, etc.

Dow

nloa

ded

11/2

8/14

to 1

29.1

20.2

42.6

1. R

edis

trib

utio

n su

bjec

t to

SIA

M li

cens

e or

cop

yrig

ht; s

ee h

ttp://

ww

w.s

iam

.org

/jour

nals

/ojs

a.ph

p



To carry out the recursive computational task, we consider a class of asynchronousand distributed algorithms in the following setup. Suppose that the state x ∈ R

r andthat there are r processors participating in the computational task. For notationalsimplicity, we assume that each processor handles only one component. It is clear thatthis can be made substantially more general by allowing each processor to handle avector of possibly different dimensions. However, the mathematical framework willbe essentially the same albeit the complex notation. Suppose that for each i ≤ r,{Y i

n} is a sequence of positive integer-valued random variables (assuming the randomsequence to be positive integer-valued is for notational convenience) that are generallystate and data dependent such that the nth iteration of processor i takes Y i

n−1 unitsof time. Define a sequence of “renewal-type” random computation times τ in as

(2.1) τ i0 = 0, τ in+1 = τ in + Y in.

For each i, the sequence {Y in} is an interarrival time and {τ in} is the corresponding

“renewal” time. It is well known that αn is strongly Markov, so ατ inis a Markov

chain.Using constant step-size μ > 0, we consider the asynchronous algorithm

(2.2) xiτ in+1

= xiτ in+μ[Mτ i

n(ατ i

n)xτ i

n]i + μ[Wτ i

n(ατ i

n)ξin]

i + μW iτ in(xτ i

n, ατ i

n, ζin), i ≤ r,

where ξin ∈ Rr and ζin ∈ R

r are the noise sequences incurred in the (n+1)st iteration.Note that the functions involved are time dependent. We use the same idea as inthe setup of a fixed configuration in the appendix, but allow more general structure.Note also that for each n and α ∈ M, Mn(α) is not a generator of a Markov chainas the fixed M discussed in the appendix. We allow the nonadditive noise to beused. When Mn(ι) = M and Wn(ι) = W are constant matrices being generators of

continuous Markov chains for all n and all ι ∈ M, and Wn ≡ 0, (2.2) reduces to theexisting standard consensus algorithm with additive noise. The nonadditive portionis a general nonlinear function of the analog state x, the Markov chain state ι ∈ M,the noise source ζ, as well as n.

The noise sequences are “exogenous” in that (loosely speaking) the distribution oftheir future evolution, conditioned on their past, does not change if we also conditionon the past of the state values. The distribution of the computation interval Y i

n is

allowed to depend on the state xτ inand the noise ξin that is used during that (n+1)st

interval in the ith processor. We define

(2.3)

Ni(n) = sup{j : τ ij ≤ n}, Δin = n− τ iNi(n)

,

ξin = ξij ζin = ζij for n ∈ [τ ij , τij+1),

xn = (x1n, . . . , xrn), where xin

def= (x1τ i

Ni(n), . . . , xrτ i

Ni(n))′,

and with a slight abuse of notation, denote αn = ατ iNi(n)

. Note that Δin = 0 if n

is a renewal time for processor i (Δin is the time elapsed since the start of a new

computation for processor i). The xn is an aggregate vector of dimension r · r and xinis the state value used for the ith processor at real time n. We can now write(2.4)

xin+1 = xin + μ[Mn(αn)xin]

iIin+1 + μ[Wn(αn)ξin]

iIin+1 + μW in(x

in, αn, ζ

in)I

in+1,

where Iin = I{Δin=0}. If Iin = 1, then n is a random computation time for the ith

processor. Note that the dependence of ατ iNi(n)

is only through the random compu-

tation time τ ij with j = Ni(n). Thus in the notation of αn, we suppressed the i

Dow

nloa

ded

11/2

8/14

to 1

29.1

20.2

42.6

1. R

edis

trib

utio

n su

bjec

t to

SIA

M li

cens

e or

cop

yrig

ht; s

ee h

ttp://

ww

w.s

iam

.org

/jour

nals

/ojs

a.ph

p



dependence for notational simplicity in the subsequent calculation; this informationis also reflected from Iin. In what follows, for each i, the mixing measures defined in(A2), namely, φ and ψ, should be i dependent. Nevertheless, to simplify the notation,instead of writing φi and ψi, we will use φ and ψ throughout the rest of the paper.We assume that the following conditions hold:

(A1) The process αn is a discrete-time Markov chain with a finite state spaceM = {1, . . . ,m0} and transition probability matrix

(2.5) P = P ε = I + εQ,

where ε > 0 is a small parameter, I is an m0 × m0 identity matrix, andQ = (qij) ∈ R

m0×m0 is the generator of a continuous-time Markov chain,(i.e., Q satisfies qij ≥ 0 for i �= j,

∑m0

j=1 qij = 0 for each i = 1, . . . ,m0) suchthat Q is irreducible.

(A2) (a) (i) The function Wn(·, ι, e) is continuous for each ι ∈ M, e ∈ Rr,

and n, and |Wn(x, ι, e)| ≤ K(1 + |x|) for each x ∈ Rr, ι ∈ M, e,

and n. (ii) The {ξin} is a sequence of Rr-valued φ-mixing processes

such that Eξin = 0, E|ξin|2+δ < ∞ for some δ > 0. Denote F ˜ξi

n =

σ{ξik; k ≤ n}, F ˜ξi,n = σ{ξik; k ≥ n}. For m > 0, the mixing mea-

sure is defined by φ(m) = supB∈F ˜ξi,n+m |P (B|F ˜ξi

n ) − P (B)| 2+δ1+δ

and

satisfies∑∞

m=1 φδ

1+δ (m) < ∞. (iii) The {ζin} is a stationary sequencethat is uniformly bounded such that for each x ∈ R

r, ι ∈ M, and n,EWn(x, ι, ζ

in) = 0. Moreover, Wn(x, ι, ζ

in) is uniformly mixing such that

the mixing measure satisfies∑

m ψ1/2(m) < ∞. (iv) The sequences

{αn}, {ξin}, and {ζin} are independent.(b) For each i, the sequence of positive integer-valued random variables {Y i

n}is bounded. There are πi(x, ι, ξi) continuous in x uniformly in each

bounded (x, ξi) set such that Eτ inY in = πi(xτ i

n, ι, ξin), where E

im denotes

the conditional expectation on F im = {x0, ατ i

j, ξij−1, ζ

ij−1 : j ≤ m}.

There are continuous and strictly positive πi(·, ι) such that for each x

and m,∑n+m

j=m Eimπ

i(x, ι, ξij)/n → πi(x, ι) in probability for each x andι ∈ M.

(c) For each ι ∈ M, {Mn(ι)} and {Wn(ι)} are uniformly bounded. For each

ι ∈ M, there is an M(ι) such that for each m,∑m+n−1

j=m Mj(ι)/n →M(ι), whereM(ι) is an irreducible generator of a Markov chain for eachι ∈ M.

Remark 2.1. (a) Note that as a consequence of (A2), for any positive integer mand fixed ι,

(2.6)

1

n

m+n−1∑j=m

EimWj(x, ι, ζ

ij) → 0 in probability,

1

n

m+n−1∑j=m

Eimξ

ij → 0 in probability.

In what follows, we often work with ξin and ζin. Then we use Eim to denote the

conditioning on the σ-algebra F im = {x0, αj , ξ

ij−1, ζ

ij−1 : j ≤ m}.

Dow

nloa

ded

11/2

8/14

to 1

29.1

20.2

42.6

1. R

edis

trib

utio

n su

bjec

t to

SIA

M li

cens

e or

cop

yrig

ht; s

ee h

ttp://

ww

w.s

iam

.org

/jour

nals

/ojs

a.ph

p



(b) It is reasonable to assume that πi(·, ι) is strictly positive. This is essentiallya suitably scaled limit of the mean of Y i

n. Under the standard renewal setup withindependent and identically distributed (i.i.d.) interarrival Y i

n (independent of data),it is simply a positive constant, the mean of Y i

n.

(c) Note that (2.4) is an SA-type algorithm, but more difficult to analyze becauseof the switching topologies. In the traditional setup of SA problems, the limit orthe averaged system is an ordinary differential equation (ODE). Very often theselimits are autonomous. Even if they are time inhomogeneous ODEs, these equationsare nonrandom. In certain cases treated here, the limit is no longer an ODE, buta randomly varying ODE subject to switching. In the literature of SA, the rate ofconvergence study is normally associated with a limit stochastic differential equation(SDE). In our case, some of the limits are Markovian-switching SDEs (i.e., switchingdiffusions [39]).

In the next two sections, three possibilities concerning the relative sizes of ε andμ are analyzed. This idea also appears in related treatments of LMS (least meansquares)-type algorithms under regime-switching dynamic systems; see [30, 31, 32].In treating the three different cases, careful analysis is needed to examine convergence,stability, and related consensus issues.

3. Asymptotic properties: ε = O(μ). This section concentrates on the caseε = O(μ). For notational simplicity and concreteness, in what follows, we simplyconsider ε = μ in this section. More general cases can be considered; they do not addfurther technical difficulties. The results will be similar in spirit.

3.1. Basic properties. To proceed, we first present a moment estimate forthe recursive algorithm (2.4). Throughout the paper, we use K to denote a genericpositive constant with the conventions K +K = K and KK = K. We also use KT

to denote a generic positive constant that depends on T (whose value may change fordifferent appearances).

Lemma 3.1. Under assumption (A1), for any 0 < T <∞ and each i = 1, . . . , r,

sup0≤n≤T/ε

E|xin|2 ≤ KT exp(T ) <∞.

Proof. Note that for any 0 < T < ∞ and 0 ≤ n ≤ T/μ, by the Cauchy–Schwarzinequality,

(3.1)

μ2E∣∣∣ n∑k=0

[Mk(αk)xik]

iIik+1

∣∣∣2 ≤ μ2(n+ 1)E

n∑k=0

∣∣∣Mk(αk)∣∣∣2∣∣∣[xik]iIik+1

∣∣∣2≤ KTμ

n∑k=0

E∣∣∣[xik]iIik+1

∣∣∣2,where KT > 0. Likewise,

(3.2)

μ2E∣∣∣ n∑k=0

Wk(xik, αk, ζ

ik)∣∣∣2 ≤ KTμ

n∑k=0


∣∣∣2 +KT ,

μ2E∣∣∣ n∑k=0

[Wk(αk)ξik]

iIik+1

∣∣∣2 ≤ K(μn)2 ≤ KT .

Dow

nloa

ded

11/2

8/14

to 1

29.1

20.2

42.6

1. R

edis

trib

utio

n su

bjec

t to

SIA

M li

cens

e or

cop

yrig

ht; s

ee h

ttp://

ww

w.s

iam

.org

/jour

nals

/ojs

a.ph

p



Iterating on E|xin|2 with the use of (2.4) and using (3.1) and (3.2), we obtain

(3.3)

E|xin+1|2 ≤ (E|xi0|2 +KT ) +KTμ

n∑k=0


∣∣∣2≤ (E|xi0|2 +KT ) +KTμ

n∑k=0

E|xik|2 +O(μ).

Then by Gronwall’s inequality,

E|xin+1|2 ≤ KT exp(nμ) ≤ KT exp(μ(T/μ)) ≤ KT exp(T ).

Taking sup over n, the desired estimate follows.

3.2. Convergence. This section is devoted to obtaining asymptotic propertiesof algorithm (2.4). Before proceeding further, we state a result on estimation errorbounds. The proof of the assertion on probability distributions is essentially in thatof Theorems 3.5 and 4.3 of [36], whereas the proof of weak convergence of αε(·) canbe found in [38]; see also [37]. Thus the proof is omitted.

Lemma 3.2. Under condition (A2), with P ε given by (2.5), denote the n-steptransition probability by (P ε)n and pεn = (P (αn = 1), . . . , P (αn = m0)), and defineαε(t) = αn for t ∈ [nε, nε+ ε). Then the following claims hold:

(3.4)pεn = p(t) +O(ε+ e−k0t/ε),

(P ε)n−n0 = Ξ(εn, εn0) +O(ε+ e−k0(n−n0)),

where p(t) ∈ R1×m0 and Ξ(t, t0) ∈ R

m0×m0 are the continuous-time probability vectorand transition matrix satisfying

(3.5)

dp(t)

dt= p(t)Q, P (0) = p0,

Ξ(t, t0)

dt= Ξ(t, t0)Q, Ξ(t0, t0) = I,

with t0 = εn0 and t = εn.Moreover, αε(·) converges weakly to α(·), a continuous-time Markov chain gen-

erated by Q.Since we consider ε = O(μ), without loss of generality, we take ε = μ in what

follows. The next lemma concerns the property of the algorithm as μ → 0 throughan appropriate continuous-time interpolation. We define

xμ(t) = xn, αμ(t) = αn for t ∈ [μn, μn+ μ).

Then (xμ(·), αμ(·)) ∈ D([0, T ] : Rr × M), which is the space of functions that aredefined on [0, T ] taking values in R

r × M, and that are right continuous and haveleft limits endowed with the Skorohod topology [14, Chapter 7]. Before proceedingfurther, we first state a lemma that gives the uniqueness of the solution of (3.6).

Lemma 3.3. The switched ODE

(3.6)dxi(t)

dt=

[M(α(t))x(t)]i

πi(x(t), α(t)), i = 1, . . . , r,

has a unique solution for each initial condition (x(0), α(0)) with x(0) = (x10, . . . , xr0)

′.

Dow

nloa

ded

11/2

8/14

to 1

29.1

20.2

42.6

1. R

edis

trib

utio

n su

bjec

t to

SIA

M li

cens

e or

cop

yrig

ht; s

ee h

ttp://

ww

w.s

iam

.org

/jour

nals

/ojs

a.ph

p



Proof. For any f(·, ·) : Rr × M → R satisfying, for each ι ∈ M, f(·, ι) ∈ C10

(space of continuously differentiable functions with compact support), L1 is definedas

(3.7) L1f(x, ι) =

r∑i=1

∂f(x, ι)

∂xi

[M(ι)x]i

πi(x, ι)+Qf(x, ·)(ι), ι ∈ M,

where

Qf(x, ·)(ι) =m0∑=1

qιf(x, �).

Let (x(t), α(t)) be a solution of the martingale problem with operator L1 definedin (3.7). We proceed to show that the solution is unique in the sense of distribution.Define

g(x, k) = exp(γ′x+ γ0k) for all γ ∈ Rr, γ0 ∈ R, k ∈ M.

Consider ψjk(t) = E[I{α(t)=j}g(x(t), k)], j, k ∈ M. It is readily seen that ψjk(t) is thecharacteristic function associated with (x(t), α(t)). By virtue of the Dynkin’s formula,

(3.8) ψj0k0(t)− ψj0k0(0)−∫ t

0

L1ψj0k0(s)ds = 0,

where

(3.9) L1ψj0k0(s) =

m0∑i=1

γi[M(k0)x]

i

πi(x(s), α(s))ψj0k0(s) +

m0∑j=0

qjj0ψjk0 (s).

Let ψ(t) = (ψι(t) : ι ≤ m0, � ≤ m0). Combining (3.8) and (3.9), we obtain

(3.10) ψ(t) = ψ(0) +

∫ t

0

Gψ(s)ds,

where G is an m0×m0 matrix. Thus (3.10) is an ODE with an initial condition ψ(0).As a result, it has a unique solution.

By Lemmas 3.1–3.3, we can obtain the following theorem.Theorem 3.4. Under (A1) and (A2), {xμ(·), αμ(·)} is tight in D([0, T ] : Rr ×

M). Assume that x0 and α0 are independent of μ and are nonrandom without lossof generality. Then (xμ(·), αμ(·)) converges weakly to (x(·), α(·)), which is a solutionof (3.6) with initial condition (x0, α0).

Proof. (a) Tightness. The tightness of {αμ(·)} can be proved as in the proofof [36, Theorem 4.3]. So we need only prove the tightness of {xμ(·)}; i.e., we needonly prove that {xμ,i(·)} is tight for each i.

For any δ > 0, let t > 0 and s > 0 such that s ≤ δ, and let t, t+ δ ∈ [0, T ]. Notethat

xμ,i(t+ s)− xμ,i(t) = μ

(t+s)/μ−1∑k=t/μ

[Mk(αk)xik]

iIik+1 + μ


[Wk(αk)ξik]

iIik+1

+ μ


[W ik(x

ik, αk, ζ

ik)]I

ik+1.

Dow

nloa

ded

11/2

8/14

to 1

29.1

20.2

42.6

1. R

edis

trib

utio

n su

bjec

t to

SIA

M li

cens

e or

cop

yrig

ht; s

ee h

ttp://

ww

w.s

iam

.org

/jour

nals

/ojs

a.ph

p



As above, hereafter we use the conventions that t/μ and (t + s)/μ denote the cor-responding integer parts, i.e., �t/μ� and �(t + s)/μ�, respectively. For notationalsimplicity, in what follows we will not use the floor function notation unless it isnecessary.

Since αk is a finite state Markov chain, by (A1) |Mk(αk)| and |Wk(αk)| (see thenotation αk in (2.3)) are uniformly bounded. Using the Cauchy–Schwarz inequality

as in (3.1) (with∑n

k=0 replaced by∑(t+s)/μ−1

k=t/μ ), together with Lemma 3.1, we get

(3.11)

Eμt

∣∣∣xμ,i(t+ s)− xμ,i(t)∣∣∣2

≤ Kμ2Eμt

⎡⎣∣∣∣∣∣(t+s)/μ−1∑

k=t/μ

[Mk(αk)xik]

iIik+1

∣∣∣∣∣2

+

∣∣∣∣∣(t+s)/μ−1∑

k=t/μ

[Wk(αk)ξik]

iIik+1

∣∣∣∣∣2⎤⎦

+Kμ2Eμt

∣∣∣∣∣(t+s)/μ−1∑

k=t/μ

[Wk(xik, αk, ζ

ik)]

iIik+1

∣∣∣∣∣2

≤ Kμs


supt/μ≤k≤(t+s)/μ−1

Eμt |xik|2 +Ks2 ≤ Kδ2,

where Eμt denotes the conditioning on Fμ

t = σ{xμ0 , ξik, ζik : i ≤ r, k < �t/μ�}. In theabove, we have used Eμ

t |xk|2 < ∞ for �t/μ� ≤ k < �(t + s)/μ�, which can be shownas in Lemma 3.1. As a result,

limδ→0

lim supμ→0

E

[sup

0≤s≤δEμ

t |xμ,i(t+ s)− xμ,i(t)|2]= 0.

The tightness of {xμ,i(·)} follows from [11, p. 47].(b) Characterization of the limit. For notational simplicity, we shall not use a

function f(·, ·) ∈ C20 in the usual martingale problem formulation for the following

derivation, but work with the underlying sequences directly. It is convenient to pro-ceed with a scaling argument to treat the random renewal times.

Define the process Zμ,in , Zμ,i(·), Ψμ,i

n , and Ψμ,i(·) by

Zμ,in = μ

n−1∑j=0

Y ij , Zμ,i(t) = Zμ,i

n on [nμ, nμ+ μ),

Ψμ,in = μ

n−1∑k=0

[Mτ ik(ατ i

k)xτ i

k]i + μ

n−1∑k=0

Wτ ik(ατ i

k)ξik + μ

n−1∑k=0

W iτ ik(xiτ i

k, ατ i

k, ζik),

Ψμ,i(t) = Ψμ,in for t ∈ [nμ, nμ+ μ),

Use a method similar to that in (a), we can prove that Zμ,i(·) and Ψμ,i(·) are tight.As a result, we obtain that {xμ,i(·), Zμ,i(·),Ψμ,i(·)} is tight in D([0,∞) : R3) and alllimits are uniformly Lipschitz continuous. We fix and work with a weakly convergentsubsequence, also indexed by μ, and with the limit denoted by (xi(·), Zi(·), Ψi(·)),for i ≤ r. Now we state a lemma.

Lemma 3.5. Under the conditions of Theorem 3.4, the limits of Zμ,i(·) andΨμ,i(·) satisfy

(3.12) Zi(t) =

∫ t

0

πi(x(Zi(s)), α(Zi(s)))ds

and

Dow

nloa

ded

11/2

8/14

to 1

29.1

20.2

42.6

1. R

edis

trib

utio

n su

bjec

t to

SIA

M li

cens

e or

cop

yrig

ht; s

ee h

ttp://

ww

w.s

iam

.org

/jour

nals

/ojs

a.ph

p



(3.13) Ψi(t) =

∫ t

0

[M(α(Zi(u))x(Zi(u))]idu,

respectively.Proof of Lemma 3.5. For fixed i, using an argument similar to that in [13, pp.

224–226], we can derive (3.12). The details are omitted.Next we work on Ψμ,i(·) and concentrate on the proof of (3.13). First, it is readily

seen that for any t, s > 0,

(3.14)

Ψμ,i(t+ s)−Ψμ,i(t)

= μ

m0∑=1

(t+s)/μ∑k=t/μ

{[Mτ i

k(�)xτ i

k]i + [Wτ i

k(�)ξik]

i + W iτ ik(xiτ i

k, �, ζik)

}I{α

τik=}.

Next, pick out any bounded and continuous function h(·), any positive integer κ, andany tj ≤ t for j ≤ κ; the weak convergence and Skorohod representation imply thatas μ→ 0,

(3.15)Eh(xμ(tj), α

μ(tj) : j ≤ κ)[Ψμ,i(t+ s)−Ψμ,i(t)]

→ Eh(x(tj), α(tj) : j ≤ κ)[Ψi(t+ s)−Ψi(t)].

Choose mμ so that mμ → ∞ as μ → 0, but μmμ = δμ → 0. By the continuity ofthe linear function in the variable x,(3.16)

limμEh(xμ(tj), α

μ(tj) : j ≤ κ)

[μ

m0∑=1

(t+s)/μ∑k=t/μ

[Mτ ik(�)xτ i

k]iI{α

τik=}

]

= limμEh(xμ(tj), α

μ(tj) : j ≤ κ)

[m0∑=1

t+s∑lδμ=t

δμmμ

lmμ+mμ−1∑k=lmμ

[Mτ ik(�)xτ i

k]iI{α

τik=}

]


μ(tj) : j ≤ κ)

[m0∑=1

t+s∑lδμ=t

δμmμ


[Mτ ik(�)xτ i

lmμ]iI{α

τik=}

]


μ(tj) : j ≤ κ)

[m0∑=1

t+s∑lδμ=t

δμmμ


[M(�)xτ ilmμ

]iI{ατik=}

]

+ limμEh(xμ(tj), α

μ(tj) : j ≤ κ)

[m0∑=1

t+s∑lδμ=t

δμmμ


[(Mτ ik(�)−M(�))xτ i

lmμ]i

×I{ατik=}

].

Denoting P (ατ ik= �|ατ i

lmμ) = p(τ ik, τ

ilmμ

) with � suppressed, inserting conditional

expectation and using a partial summation, we obtain that

(3.17)

1

mμ


(Mτ ik(�)−M(�))p(τ ik, τ

ilmμ

)

=1

mμ


(Mτ ik(�)−M(�))p(τ ilmμ+mμ−1, τ

ilmμ

)

+1

mμ


j∑k=lmμ

(Mτ ik(�)−M(�))[p(τ ij , τ

ilmμ

)− p(τ ij+1, τilmμ

)].

Dow

nloa

ded

11/2

8/14

to 1

29.1

20.2

42.6

1. R

edis

trib

utio

n su

bjec

t to

SIA

M li

cens

e or

cop

yrig

ht; s

ee h

ttp://

ww

w.s

iam

.org

/jour

nals

/ojs

a.ph

p



By virtue of the assumption (A2)(c), the term in the second line of (3.17) goes to 0as μ→ 0. Next, noting

(I + μQ)τ ij+1−τ i

lmμ − (I + μQ)τ ij−τ i

lmμ = O(μ),

we have

[p(τ ij , τilmμ

)− p(τ ij+1, τilmμ

)] = O(μ).

As a result,

E

∣∣∣∣∣h(xμ(tj), αμ(tj) : j ≤ κ)Eτ ilmμ

[m0∑=1

t+s∑lδμ=t

δμmμ



lmμ]i

×I{ατik=}

]∣∣∣∣∣≤ E

∣∣∣∣∣m0∑=1

t+s∑lδμ=t

δμmμ



lmμ]iI{α

τik=}

∣∣∣∣∣≤

m0∑=1

t+s∑lδμ=t

∣∣∣∣∣ δμmμ


(Mτ ik(�)−M(�))

∣∣∣∣∣E|xτ ilmμ

|O(μ)

→ 0 as μ→ 0.

Since μmμ → 0 as μ → 0, when μlmμ → u, for all lmμ ≤ k ≤ lmμ +mμ, μk → uas well. Therefore, the detailed estimates above lead to(3.18)

limμEh(xμ(tj), α

μ(tj) : j ≤ κ)

[μ

m0∑=1

(t+s)/μ∑k=t/μ

[Mτ ik(�)xτ i

lmμ]iI{α

τik=}

]


μ(tj) : j ≤ κ)

[m0∑=1

t+s∑lδμ=t

δμmμ

[M(�)xτ ilmμ

]ilmμ+mμ−1∑

k=lmμ

I{ατik=}

]


μ(tj) : j ≤ κ)

[m0∑=1

t+s∑lδμ=t

δμmμ


[M(ατ ik)xτ i

lmμ]i

]


μ(tj) : j ≤ κ)

[m0∑=1

t+s∑lδμ=t

δμ

× 1

mμ


[M(αμ(Zμ,i(μlmμ)))xμ(Zμ,i(μlmμ))]

i

]

= Eh(x(tj), α(tj) : j ≤ κ)

[ ∫ t+s

t

[M(α(Zi(u)))x(Zi(u))]idu

].

Next, using the independence of αn with ξin and [5, Corollary 2.4], a similarconditional expectation together with the mixing conditions given in (A2) (a) yields

(3.19)

μ

(t+s)/μ∑k=t/μ

E|Wτ ik(�)Ei

t/μξik|P (αk = �|αt/μ)

≤ Kμ

(t+s)/μ∑k=t/μ

φδ/(1+δ)(k − (t/μ)) → 0 as μ→ 0.

Dow

nloa

ded

11/2

8/14

to 1

29.1

20.2

42.6

1. R

edis

trib

utio

n su

bjec

t to

SIA

M li

cens

e or

cop

yrig

ht; s

ee h

ttp://

ww

w.s

iam

.org

/jour

nals

/ojs

a.ph

p



Likewise, using the continuity of W (·, �, ζi), the limit of

m0∑=1

μ

(t+s)/μ∑k=t/μ

Wτ ik(xiτ i

k, �, ζik)I{ατi

k=}

is the same as that of

m0∑=1

t+s∑lδμ=t

δμ1

mμ


Wτ ik(xiτ i

lmμ

, �, ζik)I{ατik=}.

Then using the uniform mixing (see [2, p. 166]) given in (A2) to the last line above,we obtain

μ

t+s∑lδμ=t

δμ1

mμ


E|Eτ ilmμ

Wτ ik(xiτ i

lmμ

, �, ζik)|P (ατ ik= �|αt/μ) → 0 as μ→ 0.

Thus,(3.20)

Eh(xμ(tj), αμ(tj) : j ≤ κ)

t+s∑lδμ=t

δμ1

mμ


Wτ ik(xiτ i

lmμ

, �, ζik)I{ατik=} → 0.

Using the estimates obtained thus far together with (3.14), we have proved that(3.21)

Eh(x(tj), α(tj) : j ≤ κ)[Ψi(t+ s)−Ψi(t)−

∫ t+s

t

[M(α(Zi(u)))x(Zi(u))]idu]= 0.

Therefore, the proof of the lemma is concluded.Completion of the proof of Theorem 3.4. With Z−1 denoting the inverse of Z, we

have(3.22)

xμ,i(t)− xi(0) = μ

Ni(t/μ)−1∑k=0

{[Mk(�)x

ik]

i + [Wk(�)ξik]

i + Wk(xik, �, ζ

ik)}Iik+1I{αk=}

= μ

(Zμ,i)−1(t)∑μk=0

{[Mk(�)xk]

i + [Wk(�)ξik]

i + Wk(xik, �, ζ

ik)}Iik+1I{αk=}.

Lemma 3.5 then yields the limit process

xi(t) = xi(0) +

∫ (Zi)−1(t)

0

[M(α(Zi(u)))x(Zi(u))]idu.

This in turn implies

xi(t) = [M(α(Zi((Zi)−1(t))))(x(Zi((Zi)−1(t))))]i(Zi)−1(t)

=[M(α(t))x(t)]i

πi(x(t), α(t))

as desired. Thus the theorem is proved.

Dow

nloa

ded

11/2

8/14

to 1

29.1

20.2

42.6

1. R

edis

trib

utio

n su

bjec

t to

SIA

M li

cens

e or

cop

yrig

ht; s

ee h

ttp://

ww

w.s

iam

.org

/jour

nals

/ojs

a.ph

p



3.3. Invariance theorem. To study the long-time behavior, we derive an in-variant theorem for the switched system. Following the discussion in [39, Chapter9], recall that a Borel measurable set U ⊂ R

r × M is invariant with respect to theprocess (x(t), α(t)) if P ((x(t), α(t)) ∈ U for all t ≥ 0) = 1 for any initial (x, ι) ∈ U .That is, a process starting from U will remain in U with probability 1. We also needthe notion of stability of sets in probability. They are defined naturally as follows:

• A closed and bounded set Kc ⊂ Rr is said to be stable in probability if

for any δ > 0 and ρ > 0, there is a δ1 > 0 such that starting from (x, ι),P (supt≥0 d(x(t),Kc) < ρ) ≥ 1− δ whenever d(x,Kc) < δ1;

• a closed and bounded set Kc ⊂ Rr is said to be asymptotically stable in

probability if it is stable in probability, and P (limt→∞ d(x(t),Kc) = 1) → 1,as d(x,Kc) → 0.

In the above, we have used the usual distance function d(x,D) = inf(|x− y| : y ∈ D).We proceed to obtain the following result.

Theorem 3.6. Assume that for each ι ∈ M, M(ι) is irreducible. Under theconditions of Theorem 3.4, the following assertions hold:

(i) The set Z = span{1l} is an invariant set.(ii) The set Z is asymptotically stable in probability.

Proof. To prove (i), we divide the time intervals according to the associated switchtimes. We begin with (x(0), α(0)) = (x0, ι). Following the dynamic system given in(3.6), let ρ1 be the first switching time, i.e., ρ1 = inf{t : α(t) = ι1 �= ι}. Note thatx(t) = x(t, ω), where ω ∈ Ω is the sample point. Then in the interval [0, ρ1), foralmost all ω, (3.6) is a system with constant matrix M(ι). For all t ∈ [0, ρ1], from(3.6) we have that for any T <∞ and t ∈ [0, T ],

x(t) =

∞∑k=0

tk

k!

dkx(0)

dtkand sup

0≤t≤T

∣∣∣dkx(0)dtk

∣∣∣ ≤ K <∞.

Because the matrix M(ι) is irreducible, there is an eigenvalue 0, and the rest of theeigenvalues all have negative real parts.

If x(0) ∈ Z, then x(0) = c1l, [M(ι)x(0)]i = 0, and, for each i ≤ r,

dxi

dt(0) =

[M(ι)x(0)]i

πi(x(0), ι)= 0,

and similarly, we obtain

dkxi

dtk(0) = 0 for all k > 0, i ≤ r.

Therefore, x(t) = x(0) for all t ∈ [0, ρ1). Thus x(t) ∈ Z for all t ∈ [0, ρ1). Now, defineρ2 = inf{t ≥ ρ1 : α(t) = ι2 �= ι1}. By the continuity of x(·), x(ρ1) = x(ρ−1 ) ∈ Z.Similarly as in the previous paragraph, we can show that for all t ∈ [ρ1, ρ2), x(t) ∈ Z.Continue in this way. For any T > 0, consider [0, T ]. Then 0 < ρ1 < ρ2 < · · · <ρN(T ) ≤ T , where N(t) is the counting that counts the number of switchings in theinterval [0, T ], and ρn is defined recursively such that α(ρn) = ιn and ρn+1 = inf{t ≥ρn : α(t) = ιn+1 �= ιn}. Suppose that we have, for all t ≤ ρn, x(t) ∈ Z with probability1 (w.p.1). Using induction, we can show x(t) ∈ Z for all t ∈ [0, ρN(T )). Finally, wework with the interval [ρN(T ), T ]; this establishes the first assertion.

Dow

nloa

ded

11/2

8/14

to 1

29.1

20.2

42.6

1. R

edis

trib

utio

n su

bjec

t to

SIA

M li

cens

e or

cop

yrig

ht; s

ee h

ttp://

ww

w.s

iam

.org

/jour

nals

/ojs

a.ph

p



To prove (ii), define V (x) = x′x/2. Since V (x) is independent of the switchingcomponent,

∑m0

=1 qιV (x) = 0. Thus, for each ι ∈ M, because of the irreducibility ofM(ι),

L1V (x) =

r∑i=1

xi[M(ι)x]i

πi(x, ι)< 0 for all x �∈ Z.

The rest of the proof of the stability in probability of the set Z is similar in spirit tothat of [39, Chapter 9]. We omit the details for brevity.

Denote xc = η1l. With the above proposition, we can further obtain the followingresult as a corollary of Theorem 3.6.

Corollary 3.7. Assume the conditions of Theorem 3.6. Then for any tμ → ∞as μ → 0, xμ(· + tμ) converges to the consensus solution η1l in probability. That is,for any δ > 0, limμ→0 P (|xμ(·+ tμ)− xc| ≥ δ) = 0.

3.4. Normalized error sequences. This section is devoted to analyzing therates of variations of a scaled sequence of errors. We begin with a result on upperbounds on estimation errors in the mean square sense.

Theorem 3.8. Assume the conditions of Theorem 3.4. Then there is an Nμ suchthat E|xn|2 = O(1) for all n ≥ Nμ.

Proof. We prove the assertion by means of perturbed Liapunov function methods.Redefine V (x) = (x − xc)

′(x − xc)/2. Note that Vxi(x) = (∂/∂xi)V (x) = (xi − xic)and Vxx(x) = I is the identity matrix. Using a Taylor expansion for V (x), we have(3.23)EnV (xn+1)− V (xn)

= μr∑

i=1

(xin − xic){(M(αn)(xn − xc))

i + [(Mn(αn)−M(αn))(xn − xc)]i

+ (Mn(αn)(xin − xn))

i + [Wn(αn)ξin]

i + W in(x

in, αn, ζ

in)

+ [(Mn(αn)−M(αn))xc]i}Iin+1 +O(μ2)(V (xn) + 1).

Note that for all xn �∈ Z,∑m0

=1 xin(M(�)xn)

iIin+1I{αn=} ≤ −λV (xn) for some λ > 0.

We will use W in(x

in, αn, ζ

in) and [Wn(x

in, αn, ζ

in)]

i interchangeably in what follows. Forsome 0 < λ0 < 1, define(3.24)

V μ1 (x, n) = μ

r∑i=1

m0∑=1

∞∑j=n

λj−n0 En(x

i − xic)[(Mj(�)−M(�))(x − xc)]iI{αj=}Iij+1,

V μ2 (x, n) = μ

r∑i=1

m0∑=1

∞∑j=n

λj−n0 En(x

i − xic)[Mj(�)(xi − x)]iI{αj=}Iij+1,

V μ3 (x, n) = μ

r∑i=1

m0∑=1

∞∑j=n

En(xi − xic)[Wj(�)ξ

ij ]

iI{αj=}Iij+1,

V μ4 (x, n) = μ

r∑i=1

m0∑=1

∞∑j=n

En(xi − xic)W

ij (x

i, �, ζij)I{αj=}Iij+1,

V μ5 (x, n) = μ

r∑i=1

m0∑=1

∞∑j=n

λj−n0 En(x

i − xic)[(M(�)−M(�))xc]iI{αj=}Iij+1.

It is easily checked that

(3.25) |V μi (x, n)| = O(μ)(V (x) + 1), i = 1, . . . , 5.

Dow

nloa

ded

11/2

8/14

to 1

29.1

20.2

42.6

1. R

edis

trib

utio

n su

bjec

t to

SIA

M li

cens

e or

cop

yrig

ht; s

ee h

ttp://

ww

w.s

iam

.org

/jour

nals

/ojs

a.ph

p



Noting

μr∑

i=1

m0∑=1

∞∑j=n+1

λj−(n+1)0 En(x

in+1 − xic)[(Mj(�)−M(�))(xn+1 − xc)]

iI{αj=}Iij+1

−μr∑

i=1

m0∑=1

∞∑j=n+1

λj−(n+1)0 En(x

in − xc)[(Mj(�)−M(�))(xn − xc)]

iI{αj=}Iij+1

= O(μ2)(V (xn) + 1),

we have(3.26)

EnVμ1 (xn+1, n+ 1)− V μ

1 (xn, n)

= En[Vμ1 (xn+1, n+ 1)− V μ

1 (xn, n+ 1)] + [EnVμ1 (xn+1, n+ 1)− V μ

1 (xn, n)]

= −μr∑

i=1

En(xin − xic)[(Mj(αn)−M(αn))]xn]

iIin+1 +O(μ2)(V (xn) + 1).

Likewise, we obtain

(3.27)

EnVμ2 (xn+1, n+ 1)− V μ

2 (xn, n)

= −μr∑

i=1

En(xin − xic)[Mj(αn)(x

in − xn)]

iIin+1 +O(μ2)(V (xn) + 1),

EnVμ3 (xn+1, n+ 1)− V μ

3 (xn, n)

= −μr∑

i=1

En(xin − xic)[Wn(αn)ξ

in]

iIin+1 +O(μ2)(V (xn) + 1),

EnVμ4 (xn+1, n+ 1)− V μ

4 (xn, n)

= −μr∑

i=1

En(xin − xic)W

in(xn, αn, ζ

in)I

in+1 +O(μ2)(V (xn) + 1),

EnVμ5 (xn+1, n+ 1)− V μ

5 (xn, n)

= −μr∑

i=1

En(xi − xic)[(Mj(αn)−M(αn))xc]

iIin+1 +O(μ2)(V (xn) + 1).

Define V μ(x, n) = V (x) +∑5

l=1 Vμl (x, n). Using (3.23), (3.26), and (3.27), we

arrive at

EnVμ(xn+1, n+ 1)− V μ(xn, n) ≤ −λV (xn) +O(μ2)(V (xn) + 1).

Using (3.25), replacing V (xn) in the last line above by V μ(xn, n), taking expectation,and iterating on the resulting inequality, we arrive at(3.28)

EV μ(xn+1, n+ 1) ≤ (1− λμ)nV μ(x0, 0) + O(μ2)

n∑k=0

(1− λμ)k

+ O(μ2)

n∑k=0

(1− λμ)n−kV μ(xk, k).

Note that there is anNμ such that for all n ≥ Nμ, we can make (1−λμ)nEV μ(x0, 0) ≤O(μ). In addition,

∑nk=0(1 − λμ)kO(μ2) = O(μ) for all n ≤ O(1/μ). Using the

estimates in the above paragraph, an application of the Gronwall’s inequality yields

Dow

nloa

ded

11/2

8/14

to 1

29.1

20.2

42.6

1. R

edis

trib

utio

n su

bjec

t to

SIA

M li

cens

e or

cop

yrig

ht; s

ee h

ttp://

ww

w.s

iam

.org

/jour

nals

/ojs

a.ph

p



that EV μ(xn+1, n+ 1) ≤ O(μ). Using (3.25) again in the estimate above, we obtainEV (xn) ≤ O(μ). The desired result thus follows.

Define

(3.29) Un =xn − xc√

μand U i

n =xin − xc√

μ.

We assume that the following assumption holds:

(A3) (i) For each � ∈ M and each ζ, Wn(·, �, ζ) has continuous partial deriva-

tives with respect to x up to the second order and Wn,xx(·, �, ζ) is uniformly

bounded, where Wn,x(·) and Wn,xx(·) denote the first and second partial

derivatives with respect to x. The {Wn(xc, �, ζin} and {Wn,x(xc, �, ζ

in)} are

bounded and uniform mixing sequences with the mixing measure satisfy-ing

∑k ψ

1/2(k) < ∞. (ii) For a sequence of indicator functions {χj(A)}where A is any measurable set with respect to {αk, Y

ik−1 : i ≤ r, k ≤ j},∑m+n−1

j=m {[Mj(�) −M(�)]xc}χj(A)/√n → 0 in probability uniformly in m.

(iii) The averaging conditions in (A2) hold with fixed m replaced by Ni(Nμ).

(iv) The sets {ξin} and {ζij} for i = 1 . . . , r are mutually independent.

Note that (A3) (i) implies that we can locally linearize Wn(·) around xc,

Wn(x, �, ζ) = Wn(xc, �, ζ) + Wn,x(xc, �, ζ)(x− xc) +O(|x − xc|2),1

n

m+n−1∑j=m

EmWj,x(xc, �, ζij) → 0 in probability,

∞∑k=n

|EnWk(xc, �, ζik)| <∞.

Condition (A3) (ii) is a technical condition similar to [14, (A1.5), p. 318]. Recall thatε = μ. It is a requirement on the rates of local average for the sequence {Mj(�)}. Itis readily verified that

(3.30)

U in+1 = U i

n + μ[Mn(αn)Uin]

iIin+1 + μ[Wn,x(xc, αn, ζin)U

in]

iIin+1

+√μ[Wn(αn)ξ

in]

iIin+1 +√μ[Wn(xc, αn, ζ

in)]

iIin+1

+√μ{[Mn(αn)−M(αn)]xc}iIin+1 +O(μ3/2)O(|U i

n|2).

To proceed, define Uμ(t) = Un for any t ∈ [μ(n − Nμ), μ(n − Nμ) + μ). Undersuitable conditions, we show that {Uμ(·)} converges weakly to a switching diffusionprocess. First note that by (A3) (ii),

√μ

(t+s)/μ−1∑j=t/μ

{[Mj(αj)−M(αj)]xc}iIij+1

=√s

1√s/μ

∑∈M


{[Mj(�)−M(�)]xc}iI{αj=}Iij+1

→ 0 as μ→ 0 uniformly in t ∈ [0, T ].

Dow

nloa

ded

11/2

8/14

to 1

29.1

20.2

42.6

1. R

edis

trib

utio

n su

bjec

t to

SIA

M li

cens

e or

cop

yrig

ht; s

ee h

ttp://

ww

w.s

iam

.org

/jour

nals

/ojs

a.ph

p



Observe that by virtue of Theorem 3.8, {Un : n ≥ Nμ} is tight. It yields that

(3.31)

Uμ,i(t+ s)− Uμ,i(t) = μ


[Mj(αj)Uij ]

iIij+1

+ μ


[Wj,x(xc, αj , ζij)U

ij ]

iIij+1

+√μ


[Wj(xc, αj , ζij)]

iIij+1

+√μ


[Wj(αj)ξij ]

iIij+1 + o(1),

where o(1) → 0 in probability. The o(1) is obtained by use of the last line of (3.30),(A3), and Theorem 3.8. Using the methods presented for analyzing xμ(·), we obtainthe following lemma, whose details are omitted.

Lemma 3.9. {Uμ(·), αμ(·)} is tight in D([0, T ] : Rr ×M).

Next, for n > 0, and each ι ∈ M, define(3.32)

Bμ,i,ιn,1 =

√μ

Nμ+n−1∑j=Nμ

[Wj(xc, ι, ζij)]

iIij+1, Bμ,i,ιn,2 =

√μ

Nμ+n−1∑j=Nμ

[Wj(ι)ξij ]

iIij+1,

Bμ,i,ι1 (t) = Bμ,i,ι

n,1 , Bμ,i,ι2 (t) = Bμ,i,ι

n,2 for t ∈ [μn, μn+ μ),

Bμ,i,ιn,1 =

√μ

Nμ+n−1∑j=Nμ

[Wj(xc, ι, ζij)]

i, Bμ,i,ιn,2 =

√μ

Nμ+n−1∑j=Nμ

= [Wj(ι)ξij ]

i,

Bμ,i,ι1 (t) = Bμ,i,ι

n,1 , Bμ,i,ι2 (t) = Bμ,i,ι

n,2 for t ∈ [μn, μn+ μ).

Define also(3.33)

Zμ,in = μ

Nμ+n−1∑j=Nμ

Y ij , Z

μ,i(t) = Zμ,in for t ∈ [μn, μn+ μ),

bμ,in,1 =√μ

Nμ+n−1∑j=Nμ

[Wj(xc, αj , ζij)]

iIij+1, bμ,in,2 =√μ

Nμ+n−1∑j=Nμ

[Wj(αj)ξij ]

iIij+1,

bμ,i1 (t) = bμ,in,1, bμ,i2 (t) = bμ,in,2 for t ∈ [μn, μn+ μ)

bμ,in,1 =√μ

Nμ+n−1∑j=Nμ

[Wj(xc, ατ ij, ζij)]

i, bμ,in,2 =√μ

Nμ+n−1∑j=Nμ

[Wj(ατ ij)ξij ]

i,

bμ,i1 (t) = bμ,in,1, bμ,i2 (t) = bμ,in,2 for t ∈ [μn, μn+ μ).

Using similar methods of the martingale averaging as in Theorem 3.4, we canshow that Bμ,i,ι

l (·) converges weakly to Bi,ιl (·) = Bi,ι

l ((Zi(·))−1) for l = 1, 2. It is also

easy to see that Bi,ι1 (·) and Bi,ι

2 (·) are independent.

Theorem 3.10. Under (A1)–(A3), there are independent standard Brownian

Dow

nloa

ded

11/2

8/14

to 1

29.1

20.2

42.6

1. R

edis

trib

utio

n su

bjec

t to

SIA

M li

cens

e or

cop

yrig

ht; s

ee h

ttp://

ww

w.s

iam

.org

/jour

nals

/ojs

a.ph

p



motions wi,1(·) and wi,2(·) for i ≤ r such that the limits U i(·), i ≤ r, satisfy

(3.34) dU i =[M(α(t))U ]i

πi(xc, α(t))dt+

[σi1(α(t))dwi,1(t) + σi

1(α(t))dwi,2(t)]√πi(xc, α(t))

, i ≤ r.

Proof. The proof is similar in spirit to that of Theorem 3.4. So we will only pointout the distinct features. Using the well-known results for mixing processes (see [2]

and [5]), it is easily seen that Bμ,i,ι1 (·) and Bμ,i,ι

2 (·) converge weakly to Brownian

motions Bi,ι1 (·) and Bi,ι

2 (·), with covariance (σi1(ι))

2t and (σi2(ι))

2t, respectively. It is

also easy to see that Bi,ι1 (·) and Bi,ι

2 (·) are independent. Using the scaling argument as

in the proof of Theorem 3.4, we can show that Bμ,i,ιl (·) converges weakly to Bi,ι

l (·) =Bi,ι

l ((Zi(·))−1) for l = 1, 2. We need to prove the independence of the limit Brownianmotions. Here we use an argument similar to [13, the last few lines of p. 239 and thefirst few lines of p. 240]. Let Kμ → ∞ such that

√μKμ → 0 and let Nμ

i = Ni(Nμ).We work with

Bμ,i,ι1 (t) =

√μ

Nμi +Kμ−1∑

j=Nμi +Kμ

[Wj(xc, ι, ζij)]

i, Bμ,i,ιn,2 =

√μ

Nμi +Kμ−1∑

j=Nμi +Kμ

[Wj(ι)ξij ]

i.

For simplicity of notation, we take r = 2. We shall show that the limits of Bμ,i,ι(·)are independent. Denote Iμmn = I{Nμ

1 =n}I{Nμ2 =m}. For any bounded and continuous

function H1(·) and H2(·), we have

EH1(Bμ,1,ι1 (t))H2(B

μ,2,ι1 (t))

=∑n,m

EH1

(√μ

n+k+(t/μ)∑j=n+k

[Wj(xc, ι, ζ1j )]

1

)H2

(√μ

m+k+(t/μ)∑j=m+k

[W (xc, ι, ζ1j )]

2

)Iμmn

=∑n,m

EH1

(√μ

n+k+(t/μ)∑j=n+k

[Wj(xc, ι, ζ1j )]

1

)EH2

(√μ

m+k+(t/μ)∑j=m+k

[W (xc, ι, ζ1j )]

2

)EIμmn

+ o(1),

where o(1) → 0 as μ → 0. This, together with the arbitrariness of k and the weakconvergence, implies the independence of the limit Brownian motions. Likewise, wecan show the independence of the limit Brownian motions associated with Bμ,i,ι

2 (·).We can then show

bμ,in,1 =√μ

Nμ+n−1∑j=Nμ

[Wτ ij(xc, ατ i

j, ζij)]

i

=√μ∑ι∈M

Nμ+n−1∑j=Nμ

[W τ ij(xc, ι, ζij)]

iI{ατij=ι}.

Choose mμ, δμ etc. as in the convergence proof of the algorithm. Then(3.35)

bμ,i1 (t+ s)− bμ,i1 (t) =∑ι∈M

√δμ

1√mμ

(t+s)/δμ∑l=t/δμ

lmμ+mμ−1∑j=lmμ

[Wτ ij(xc, ι, ζ

ij)]

iI{ατij=ι}.

Dow

nloa

ded

11/2

8/14

to 1

29.1

20.2

42.6

1. R

edis

trib

utio

n su

bjec

t to

SIA

M li

cens

e or

cop

yrig

ht; s

ee h

ttp://

ww

w.s

iam

.org

/jour

nals

/ojs

a.ph

p



Let μlmμ → u. Then for all lmμ ≤ j ≤ lmμ + mμ − 1, μk → u. Using the weakconvergence of αμ(·) to α(·) and the Skorohod representation, the limit in the lastline of (3.35) is the same as that of

∑ι∈M

√δμ

1√mμ

(t+s)/δμ∑l=t/δμ

lmμ+mμ−1∑j=lmμ

[Wτ ij(xc, ι, ζ

ij)]

iI{α(u)=ι}.

Thus, the limit of (3.35) is given by

(3.36)

bi1(t+ s)− bi1(t) =∑ι∈M

∫ t+s

t

σi1(ι)I{α(u)=ι}dwi,1(u)

=

∫ t+s

t

σi1(α(u))dwi,1(u),

where wi,1(·) is a standard Brownian motion. Likewise, bμ,i2 (·) converges to a switched

Brownian motion in the sense that bi2(t) =∫ t

0 σi(α(u))dwi,2(u). Thus b

μ,i(·) = bμ,i1 (·)+bμ,i2 (·) converges weakly to Bi(·) such that

bi(t) =

∫ t

0

[σi1(α(u))dwi,1(u) + σi

2(α(u))dwi,2(u)].

The last step is to combine the above estimates, together with the independence ofthe limit Brownian motions established, with a scaling argument as in the proof ofTheorem 3.4. A few details are omitted.

Remark 3.11. In view of the independence of the Brownian motions wi,1(·) andwi,2(·), there is a standard Brownian motion wi(·) such that the switching diffusion(3.34) may also be written in an equivalent form as

(3.37) dU i =[M(α(t))U ]i

πi(xc, α(t))dt+ σi(α(t))dwi(t), i ≤ r,

where

[σi(�)]2 =[σi

1(�)]2 + [σi

2(�)]2

πi(xc, �), � ∈ M, i ≤ r.

4. Slowly varying (ε � μ) and rapidly varying (μ � ε) Markov chains.This section is divided into two subsections. One is concerned with slowly varyingMarkov chains (0 < ε � μ), whereas the other treats rapidly switching processes(0 < μ � ε).

4.1. Slowly varying Markov chains. Suppose that ε � μ, where ε is theparameter appeared in the transition probability matrix of the Markov chain and μis the step-size of the algorithm (2.4). Intuitively, because the Markov chain changesso slowly, the time-varying parameter process is essentially a constant. We reveal theasymptotic properties of the recursive algorithm. To facilitate the discussion and tosimplify the notation, we take ε = μ2 in what follows.

Note that Lemma 3.2 still holds. We next analyze algorithm (2.4). As in theprevious case, we can prove sup0≤n≤O(1/μ) E|xin|2 <∞. Define the piecewise constantinterpolation xμ(t) = xn for t ∈ [μn, μn + μ). Then as in the previous section, we

Dow

nloa

ded

11/2

8/14

to 1

29.1

20.2

42.6

1. R

edis

trib

utio

n su

bjec

t to

SIA

M li

cens

e or

cop

yrig

ht; s

ee h

ttp://

ww

w.s

iam

.org

/jour

nals

/ojs

a.ph

p



have that {xμ,i(·)} is tight in D([0, T ],R). We proceed to characterize its limit. Theanalysis is similar to that of Theorem 3.4, so we will omit most of the details.

The idea is that since the Markov chain is slowly varying. The the parameteris almost a constant. Since α0 =

∑m0

ι=1 ιI{α0=ι}, we obtain the desired result with

[M(ι)x(u)]i/π(x(u), ι) in (3.6) replaced by∑m0

ι=1 pι[M(ι)x(u)]i/πi(x(u), ι). We sum-marize the discussions above into the following result.

Theorem 4.1. Assume the conditions of Theorem 3.4 with the modification thatthe step-size in (2.4) satisfies ε = μ2. Then xμ(·) converges weakly to x(·), which isa solution of the ODE

(4.1)dxi(t)

dt=

m0∑ι=1

pι[M(ι)x(t)]i

πi(x(t), ι).

In addition, for any tμ → ∞ as μ → 0, xμ(· + tμ) converges to the consensussolution η1l in probability. That is, for any δ > 0, limμ→0 P (|xμ(·+ tε)−xc| ≥ δ) = 0.

Remark 4.2. To carry out the error analysis, furthermore, we define xc and Un asbefore and show that {Un : n ≥ Nμ} is tight. If we let Uμ(t) be a piecewise constantinterpolation of Un on t ∈ [(n−Nμ)μ, (n −Nμ)μ + μ), similar to Remark 3.11, thenUμ(·) converges weakly to U(·) such that U(·) is the solution of the SDE

dU i =

m0∑ι=1

pι[M(ι)U(t)]i

πi(xc, ι)dt+ σi(t)dwi(t),

where wi(·) is a standard Brownian motion and

[σi(t)]2 =

m0∑ι=1

pι[σi

1(ι)]2 + [σi

2(ι)]2

πi(xc, ι), i ≤ r.

Note that the interpolation of the centered and scaled sequence of errors has a diffusionlimit in which the drift and diffusion coefficients are averaged out with respect to theinitial probability distribution.

4.2. Fast changing Markov chains. This section takes up the issue that theMarkov chain is fast varying compared to the adaptation. By that, we mean μ � ε.For concreteness of the discussion, we take a specific form of the step-size, namely,ε = μ1/2. Intuitively, the Markov chain varies relatively fast and can be thought ofas a noise process. Eventually it is averaged out.

For αlmμ = i,

P{αk = j|αlmμ} = Ξij(εlmμ, εk) +O(ε+ exp(−κ)).In view of (3.5) and noting ε = μ1/2 and irreducibility of Q, we have Ξij(εlmμ, εk) =νj+O(exp(−κ0 k−lmmu√

μ )), where νj is the jth component of the stationary distribution

ν = (ν1, . . . , νm0) associated with the generator Q of the corresponding continuous-time Markov chain. This indicates that Ξ(s, t) can be approximated by a matrix 1lνwith identical rows. Thus we obtain the limit ODE.

Theorem 4.3. Assume the conditions of Theorem 3.4 with the modification thatthe step-size in (2.4) satisfies ε = μ1/2. Then xμ(·) converges weakly to x(·), which isa solution of the ODE

(4.2)dxi(t)

dt=

m0∑ι=1

νι[M(ι)x(t)]i

πi(x(t), ι).

Dow

nloa

ded

11/2

8/14

to 1

29.1

20.2

42.6

1. R

edis

trib

utio

n su

bjec

t to

SIA

M li

cens

e or

cop

yrig

ht; s

ee h

ttp://

ww

w.s

iam

.org

/jour

nals

/ojs

a.ph

p



In addition, for any tμ → ∞ as μ → 0 and for any δ > 0, limμ→0 P (|xμ(·+ tμ)−xc| ≥ δ) = 0.

Remark 4.4. Concerning the errors, for the fast changing Markov chain case,within a very short period of time, the system is replaced by an average with respectto the stationary distribution of the Markov chain. For the error analysis, furthermore,we may define xc Un as in (3.29) and show that {Un : n ≥ Nμ} is tight. If we letUμ(t) be the piecewise constant interpolation of Un on t ∈ [(n−Nμ)μ, (n−Nμ)μ+μ),similar to Remark 3.11, then Uμ(·) converges weakly to U(·) such that U(·) is thesolution of the SDE

dU i =

m0∑ι=1

νι[M(ι)U(t)]i

πi(xc, ι)dt+ σi(t)dwi(t),

where wi(·) is a standard Brownian motion and

[σi(t)]2 =

m0∑ι=1

νι[σi

1(ι)]2 + [σi

2(ι)]2

πi(xc, ι), i ≤ r.

Remark 4.5. As was mentioned, for convenience of presentation, we chose ε = μ2

and ε =√μ for the slowly varying and fast varying cases. The specific forms of μ and

ε enable us to simplify the presentation. The convergence results remain essentiallythe same for the general cases μ/ε→ 0 and μ/ε→ ∞.

5. Illustrative examples. This section presents several simulation examples.We call (xn − xc)

′(xn − xc) the consensus error variance at time n.

Example 5.1. Suppose that the Markov chain αn has only two states, i.e., M ={1, 2}. The transition probability matrix is

P ε = I + ε

( −0.4 0.40.3 −0.3

).

For a given system of five subsystems, suppose the link gains are G1 = diag(1, 0.3, 1.2,4, 7, 10) and G2 = diag(2, 0.5, 1, 6, 9, 14) with regime-switching at two different states.Suppose the initial states are x10 = 12, x20 = 34, x30 = 56, x40 = 8, x50 = 76. The stateaverage is η = 37.2 (xc = η1l). The initial consensus error is (x0 − xc)

′(x0 − xc) =3356.8. Take ε = 0.02 and step-size μ = ε = 0.02. The updating algorithm runs for3000 steps, and the stopped consensus error variance is (x3000 − xc)

′(x3000 − xc) =8.0166. In Figure 2, we plot the Markov chain state trajectories and the system statetrajectories.

Example 5.2. Here we consider the case that the Markov chain changes veryslowly compared with the adaptation step-size. That is, ε � μ. To be specific,suppose ε = μ2, where μ = 0.02. The numerical results are shown in Figure 3. Fromthe trajectory of the Markov chain, there is only one switching taking place in thefirst 1000 iterations. The convergence of the consensus is also demonstrated.

Example 5.3. Here we consider the fast changing Markov μ� ε. Specifically, wetake μ = ε2 with μ = 0.02. The corresponding trajectories are plotted in Figure 4.The frequent Markov switching is clearly seen.

6. Further remarks. For convenience and notational simplicity, we have usedthe current setup. Several extensions and generalizations can be carried out. So far,

Dow

nloa

ded

11/2

8/14

to 1

29.1

20.2

42.6

1. R

edis

trib

utio

n su

bjec

t to

SIA

M li

cens

e or

cop

yrig

ht; s

ee h

ttp://

ww

w.s

iam

.org

/jour

nals

/ojs

a.ph

p



0 500 1000 1500 2000 2500 30000.5

1

1.5

2

2.5

Iteration Number

M

0 500 1000 1500 2000 2500 30000

10

20

30

40

50

60

70

80

90

Iteration Number

(a) Markov chain sample paths.(b) System state trajectories: Values ofxn.

Fig. 2. Trajectories of the case ε = μ = 0.02. (Horizontal axes express discrete times oriteration numbers.)

0 500 1000 1500 2000 2500 30000.5

1

1.5

2

2.5

Iteration Number

M

0 500 1000 1500 2000 2500 30000

10

20

30

40

50

60

70

80

90

Iteration Number


Fig. 3. Slowly varying Markov parameter μ = 0.02 and ε = μ2. (Horizontal axes expressdiscrete times or iteration numbers.)

0 500 1000 1500 2000 2500 30000.5

1

1.5

2

2.5

Iteration Number

M

0 500 1000 1500 2000 2500 30000

10

20

30

40

50

60

70

80

Iteration Number


Fig. 4. Fast varying Markov parameter μ = 0.02 and ε =√μ. (Horizontal axes express discrete

times or iteration numbers.)

the noise sequences are correlated random processes. For convenience, we used mixing-type noise processes. All of the development up to this point can be generalized tomore complex x-dependent noise processes [14, sections 6.6 and 8.4].

Dow

nloa

ded

11/2

8/14

to 1

29.1

20.2

42.6

1. R

edis

trib

utio

n su

bjec

t to

SIA

M li

cens

e or

cop

yrig

ht; s

ee h

ttp://

ww

w.s

iam

.org

/jour

nals

/ojs

a.ph

p



To conclude, this paper provided a class of asynchronous SA algorithms forconsensus-type problems with randomly switching topologies. This study extendedthe arenas for consensus-type control problems to randomly time-varying dynamicsof networked systems.

Appendix. Consensus algorithm basics: Traditional setting. This appen-dix gives a brief account of the setup of consensus under simple conditions. Considera networked system of r nodes, given by

(A.1) xin+1 = xin + uin, i = 1, . . . , r,

where uin is the node control for the ith node, or in a vector form, xn+1 = xn+un withxn = [x1n, . . . , x

rn]

′, un = [u1n, . . . , urn]

′. The nodes are linked by a sensing network,represented by a directed graph G whose element (i, j) indicates estimation of thestate xjn by node i via a communication link, and a permitted control vijn on the link.For node i, (i, j) ∈ G is a departing edge and (l, i) ∈ G is an entering edge. The totalnumber of communication links in G is ls. From its physical meaning, node i canalways observe its own state, which will not be considered as a link in G.

We consider link controls among nodes permitted by G. The node control uin isdetermined by the link control vijn . Since a positive transportation of quantity vijn on(i, j) means a loss of vijn at node i and a gain of vijn at node j, the node control atnode i is uin = −∑(i,j)∈G v

ijn +

∑(j,i)∈G v

jin . The most relevant implication in this

control scheme is that for all n,∑r

i=1 xin =

∑ri=1 x

i0 := ηr for some η ∈ R that is the

average of x0. That is, η =∑r

i=1 xi0/r. Consensus control seeks control algorithms

that achieve xn → η1l, where 1l is the column vector of all 1s. A link (i, j) ∈ G entailsan estimate, denoted by xijn , of x

jn by node i with estimation error dijn , i.e.,

(A.2) xijn = xjn + dijn .

The estimation error dijn is usually a function of the signal xjn itself and depends oncommunication channel noises ξijn in a nonadditive and nonlinear relation

(A.3) dijn = g(xjn, ξijn )

and can be spatially and temporally dependent. Most existing literature considersmuch simplified noise classes dijn = ξijn with i.i.d. assumptions.

Such extensions are necessary when dealing with networked systems. A sampledand quantized signal x in a networked system enters a communication transmitter asa source. To enhance channel efficiency and reduce noise effects, source symbols areencoded [6, 18]. Typical block or convolutional coding schemes such as Hamming,Reed–Solomon, or, more recently, the low-density parity-check (LDPC) code andTurbo code, often introduce a nonlinear mapping v = f1(x). The code word v is thenmodulated into a waveform s = f2(v) = f2(f1(x)) which is then transmitted. Evenwhen the channel noise is additive, namely, the received waveform is w = s+d where dis the channel noise, after the reverse process of demodulation and decoding, we havey = g(w) = g(s+d) = g(f2(f1(x))+d). As a result, the error term g(f2(f1(x))+d)−xin general is nonadditive and signal dependent. In addition, block and convolutioncoding schemes introduce temporally dependent noises. In our formulation, this aspectis reflected in dependent φ-mixing noises on ξijn . These will be detailed later.

For simplification on system derivations, we use first dijn = ξijn in this section. Letηn and ξn be the ls-dimensional vectors that contain all xijn and ξijn in a selected order,

Dow

nloa

ded

11/2

8/14

to 1

29.1

20.2

42.6

1. R

edis

trib

utio

n su

bjec

t to

SIA

M li

cens

e or

cop

yrig

ht; s

ee h

ttp://

ww

w.s

iam

.org

/jour

nals

/ojs

a.ph

p



respectively. Then, (A.2) can be written as ηn = H1xn + ξn, where H1 is an ls × r

matrix whose rows are elementary vectors such that if the �th element of ζn is xij , thenthe �th row in H1 is the row vector of all zeros except for a “1” at the jth position.Each sensing link provides information δijn = xin−xijn , an estimated difference betweenxin and xjn. This information may be represented, in the same arrangement as ηn, bya vector δn of size ls containing all δijn in the same order as ηn. δn can be writtenas δn = H2xn − ηn = H2xn −H1xn − ξn = Hxn − ξn, where H2 is an ls × r matrixwhose rows are elementary vectors such that if the �th element of ζ(k) is xij , then the�th row in H2 is the row vector of all zeros except for a “1” at the ith position, andH = H2 −H1. The reader is referred to [3] for basic matrix properties in graphs andto [27] for matrix iterative schemes. Due to network constraints, the information δijncan only be used by nodes i and j. When the control is linear, time invariant, andmemoryless, we have vijn = μgijδ

ijn , where gij is the link control gain on (i, j) and μ is

a global scaling factor that will be used in state updating algorithms as the recursivestep-size. Let G be the ls × ls diagonal matrix that has gij as its diagonal element.In this case, the node control becomes un = −μH ′Gδn. For convergence analysis,we note that μ is a global control variable, and we may represent un equivalently asun = −μ(H ′GHxn −H ′Gξn) = μ(Mxn +Wξn), with M = −H ′GH and W = H ′G.

Under the link-based state control uin, the state updating scheme (A.1) becomes

(A.4) xn+1 = xn − μH ′Gδn.

Since 1l′M = 0, 1l′W = 0, 1l′xn+1 = 1l′xn = rη hold for all n, which is a naturalconstraint to the SA algorithm. Starting at x0, xn is updated iteratively by using(A.4), which for the analysis is

(A.5) xn+1 = xn + μ(Mxn +Wξn).

Throughout the paper, the noise {ξn} is allowed to be correlated, both spatially andtemporally. We will assume the following conditions.

(A0) (i) All link gains are positive, gij > 0. (ii) G contains a spanning tree.

Recall that a square matrix Q = (qij) is a generator of a continuous-time Markovchain if qij ≥ 0 for all i �= j and

∑j qij = 0 for each i. Also, a generator or the

associated continuous-time Markov chain is irreducible if the system of equations{νQ = 0,ν1l = 1

has a unique solution, where ν = [ν1, . . . , νr] ∈ R1×r with νi > 0 for each i = 1, . . . , r

is the associated stationary distribution. Assume that the noise is unbounded but hasbounded (2 + Δ)th moments. In addition, it is a sequence of correlated noise, muchbeyond the usual i.i.d. noise classes. A φ-mixing sequence has the property that theremote past and the distant future are asymptotically independent. The asymptoticindependence is reflected by the condition on the underlying mixing measure. Theproof of the following theorem is in [33].

Theorem A.1. Under assumption (A0), (1) M has rank r − 1 and is negativesemidefinite, and (2) M is a generator of a continuous-time Markov chain and isirreducible.

Dow

nloa

ded

11/2

8/14

to 1

29.1

20.2

42.6

1. R

edis

trib

utio

n su

bjec

t to

SIA

M li

cens

e or

cop

yrig

ht; s

ee h

ttp://

ww

w.s

iam

.org

/jour

nals

/ojs

a.ph

p



REFERENCES

[1] I. Akyildiz, W. Su, Y. Sankarasubramniam, and E. Cayirci, A survey on sensor networks,IEEE Commun. Mag., 40 (2002), pp. 102–114.

[2] P. Billingsley, Convergence of Probability Measures, John Wiley, New York, 1968.[3] R. A. Brualdi and H. J. Ryser, Combinatorial Matrix Theory, Cambridge University Press,

Cambridge, UK, 1991.[4] J. Cortes and F. Bullo, Coordination and geometric optimization via distributed dynamical

systems, SIAM J. Control Optim., 44 (2005), pp. 1543–1574.[5] S. N. Ethier and T. G. Kurtz, Markov Processes: Characterization and Convergence, John

Wiley, New York, 1986.[6] S. Haykin, Digital Communications, 4th ed., John Wiley, New York, 2001.[7] M. Huang and J. H. Manton, Stochastic approximation for consensus seeking: Mean square

and almost sure convergence, in Proceedings of the 46th IEEE Conference on Decision andControl (CDC), New Orleans, LA, 2007, pp. 306–311.

[8] M. Huang, S. Dey, G. N. Nair, and J. H. Manton, Stochastic consensus over noisy networkswith Markovian and arbitrary switches, Automatica J. IFAC, 46 (2010), pp. 1571–1583.

[9] A. Jadbabaie, J. Lin, and A. S. Morse, Coordination of groups of mobile autonomous agentsusing nearest neighbor rules, IEEE Trans. Automat. Control, 48 (2003), pp. 988–1000.

[10] S. Kar and J. M. F. Moura, Distributed consensus algorithms in sensor networks with im-perfect communication: Link failures and channel noise, IEEE Trans. Signal Process., 57(2009), pp. 355–369.

[11] H. J. Kushner, Approximation and Weak Convergence Methods for Random Processes, withApplications to Stochastic Systems Theory, MIT Press, Cambridge, MA, 1984.

[12] H. J. Kushner and G. Yin, Asymptotic properties of distributed and communicating stochasticapproximation algorithms, SIAM J. Control Optim., 25 (1987), pp. 1266–1290.

[13] H. J. Kushner and G. Yin, Stochastic approximation algorithms for parallel and distributedprocessing, Stochastics, 22 (1987), pp. 219–250.

[14] H. J. Kushner and G. Yin, Stochastic Approximation and Recursive Algorithms and Appli-cations, 2nd ed., Springer-Verlag, New York, 2003.

[15] T. Li and J. F. Zhang, Consensus conditions of multi-agent systems with time-varying topolo-gies and stochastic communication noises, IEEE Trans. Automat. Control, 55 (2010), pp.2043–2057.

[16] Y. Liu, K. Passino, and M. M. Polycarpou, Stability analysis of M-dimensional asynchro-nous swarms with a fixed communication topology, IEEE Trans. Automat. Control, 48(2003), pp. 76–95.

[17] N. A. Lynch, Distributed Algorithms, Morgan Kaufmann, San Francisco, 1997.[18] T. K. Moon, Error Correction Coding, Mathematical Methods and Algorithms, John Wiley,

New York, 2005.[19] L. Moreau, Stability of multiagent systems with time-dependent communication links, IEEE

Trans. Automat. Control, 50 (2005), pp. 169–182.[20] R. Olfati-Saber and R. M. Murray, Consensus problems in networks of agents with switch-

ing topology and time-delays, IEEE Trans. Automat. Control, 49 (2004), pp. 1520–1533.[21] R. Olfati-Saber, J. A. Fax, and R. M. Murray, Consensus and cooperation in networked

multi-agent systems, IEEE Proc., 95 (2007), pp. 215–233.[22] P. Ogren, E. Fiorelli, and N. E. Leonard, Cooperative control of mobile sensor networks:

Adaptive gradient climbing in a distributed environment, IEEE Trans. Automat. Control,49 (2005), pp. 1292–1302.

[23] W. Ren and R. W. Beard, Consensus seeking in multiagent systems under dynamically chang-ing interaction topologies, IEEE Trans. Automat. Control, 50 (2005), pp. 655–661.

[24] C. W. Reynolds, Flocks, herds, and schools: A distributed behavioral model, Comput. Graph-ics, 21 (1987), pp. 25–34.

[25] J. Toner and Y. Tu, Flocks, herds, and schools: A quantitative theory of flocking, Phys. Rev.E, 58 (1998), pp. 4828–4858.

[26] J. N. Tsitsiklis, D. P. Bertsekas, and M. Athans, Distributed asynchronous deterministicand stochastic gradient optimization algorithms, IEEE Trans. Automat. Control, 31 (1986),pp. 803–812.

[27] R. S. Varga, Matrix Iterative Analysis, Spring-Verlag, Berlin, 2000.[28] T. Viseck, A. Czirook, E. Ben-Jacob, O. Cohen, and I. Shochet, Novel type of phase

transition in a system of self-driven particles, Phys. Rev. Lett., 75 (1995), pp. 1226–1229.[29] L. Xiao, S. Boyd, and S. J. Kim, Distributed average consensus with least-mean-square devi-

ation, J. Parallel Distrib. Comput., 67 (2007), pp. 33–46.

Dow

nloa

ded

11/2

8/14

to 1

29.1

20.2

42.6

1. R

edis

trib

utio

n su

bjec

t to

SIA

M li

cens

e or

cop

yrig

ht; s

ee h

ttp://

ww

w.s

iam

.org

/jour

nals

/ojs

a.ph

p



[30] G. Yin, C. Ion, and V. Krishnamurthy, How does a stochastic optimization/approximationalgorithm adapt to a randomly evolving optimum/root with jump Markov sample paths,Math. Programming Ser. B, 120 (2009), pp. 67–99.

[31] G. Yin, V. Krishnamurthy, and C. Ion, Regime switching stochastic approximation algo-rithms with application to adaptive discrete stochastic optimization, SIAM J. Optim., 14(2004), pp. 1187–1215.

[32] G. Yin and V. Krishnamurthy, Least mean square algorithms with Markov regime switchinglimit, IEEE Trans. Automat. Control, 50 (2005), pp. 577–593.

[33] G. Yin, Y. Sun, and L. Y. Wang, Asymptotic properties of consensus-type algorithms fornetworked systems with regime-switching topologies, Automatica J. IFAC, 47 (2011), pp.1366–1378.

[34] G. Yin, L. Y. Wang, and Y. Sun, Stochastic recursive algorithms for networked systems withdelay and random switching: Multiscale formulations and asymptotic properties, Multi-scale Model. Simul., 9 (2011), pp. 1087–1112.

[35] G. Yin, C. Z. Xu, and L. Y. Wang, Optimal remapping in dynamic bulk synchronous compu-tations via a stochastic control approach, IEEE Trans. Parallel Distrib. Systems, 14 (2003),pp. 51–62.

[36] G. Yin and Q. Zhang, Singularly perturbed discrete-time Markov chains, SIAM J. Appl. Math.,61 (2000), pp. 834–854.

[37] G. Yin and Q. Zhang, Discrete-Time Markov Chains: Two-Time-Scale Methods and Appli-cations, Springer, New York, 2005.

[38] G. Yin, Q. Zhang, and G. Badowski, Discrete-time singularly perturbed Markov chains:Aggregation, occupation measures, and switching diffusion limit, Adv. in Appl. Probab.,35 (2003), pp. 449–476.

[39] G. Yin and C. Zhu, Hybrid Switching Diffusions: Properties and Applications, Springer, NewYork, 2010.

[40] G. Yin and Y. M. Zhu, On w.p.1 convergence of a parallel stochastic approximation algorithm,Probab. Engrg. Inform. Sci., 3 (1989), pp. 55–75.

Dow

nloa

ded

11/2

8/14

to 1

29.1

20.2

42.6

1. R

edis

trib

utio

n su

bjec

t to

SIA

M li

cens

e or

cop

yrig

ht; s

ee h

ttp://

ww

w.s

iam

.org

/jour

nals

/ojs

a.ph

p

Documents

Asynchronous Stochastic Approximation Algorithms for Networked Systems: Regime-Switching Topologies and Multiscale Structure