On information transmission in linear feedback tracking systems

ASIA-PACIFIC JOURNAL OF CHEMICAL ENGINEERINGAsia-Pac. J. Chem. Eng. 2008; 3: 630–637Published online 22 September 2008 in Wiley InterScience(www.interscience.wiley.com) DOI:10.1002/apj.206

Special Theme Research Article

On information transmission in linear feedback trackingsystems†

Hui Zhang* and Youxian Sun

State Key Laboratory of Industrial Control Technology, Institute of Industrial Process Control, Department of Control Science and Engineering,Zhejiang University, Hang Zhou, China

Received 24 July 2008; Accepted 25 July 2008

ABSTRACT: Information transmission in discrete time linear time-invariant (LTI) feedback tracking systems wasinvestigated by using measures of directed information and mutual information. It was proved that, for a pair ofextraneous input and internal variable, directed information (rate) is always equal to mutual information (rate); fora pair of internal variables, the former is smaller than the latter. Furthermore, the feedback changes the informationtransmission between internal variables, while it has no influence on information transmission from extraneous variableto internal variable. Consideration on system design was discussed. 2008 Curtin University of Technology and JohnWiley & Sons, Ltd.

KEYWORDS: directed information; mutual information; linear tracking system; information transmission; feedback

INTRODUCTION

As a new measure of information transmission, theso-called directed information defined by Massey,[1]

which is different from the traditional measure ofmutual information defined by Shannon,[2] is attractingattention in the fields of information theory[3] andcontrol systems with communication constraints.[4,5]

It was demonstrated[1] that for finite states channelswith or without memory, the directed information andthe mutual information between channel input andoutput are identical if the channel is used withoutfeedback; when there is feedback from channel outputto encoder, the directed information is strictly smallerthan mutual information. The key point here is that‘causality independence’ does not mean ‘statisticalindependence’.[1,3]

On the other hand, information theoretic approachesto the analysis and design of control system (withno communication constraints) are attracting more andmore attention recently.[6–12] The attempts at investigat-ing the relation between control and information make itimportant to investigate the information transmission infeedback systems. For example, as measures concerning

*Correspondence to: Hui Zhang, State Key Laboratory of IndustrialControl Technology, Institute of Industrial Process Control, Depart-ment of Control Science and Engineering, Zhejiang University, HangZhou, 310027, China. E-mail: zhanghui [email protected]†This work is supported by the National Natural Science Foundationof China (60674028).

information transmission, entropy rate and mutual infor-mation rate play important roles in stochastic controland estimation problems.[6,8,11,12] However, the func-tion of directed information (rate), which measures thereal information transmission in causal systems, has notbeen discussed for control.

In this paper, we will investigate the relation betweenthe directed information (rate) and mutual informa-tion (rate) in linear feedback tracking control sys-tems (with no communication constraints). The samplespace of random variables is continuous. Our workslead to the conclusions that, in measuring informa-tion transmission between extraneous inputs and inter-nal variables, the directed information is always equalto the mutual information; for pairs of internal vari-ables, the former is identical with or smaller thanthe latter. Furthermore, by comparing the open- andclosed-loop systems, we understand that the feedbackchanges the information transmission between internalvariables, while it makes no influence on informationtransmission from extraneous variables to internal vari-ables. This is slightly different from the conclusion incommunication channel, which states that for contin-uous alphabet Gaussian channels with colored noise,the capacity (defined as the maximum mutual informa-tion between message and channel output) is increasedby feedback.[13] Information theoretic preliminaries andthe system under discussion will be presented in Sec-tion 2 with notations. Section 3 will give the mainresults. Section 4 will be the conclusion and discus-sion.

2008 Curtin University of Technology and John Wiley & Sons, Ltd.

Asia-Pacific Journal of Chemical Engineering ON INFORMATION TRANSMISSION 631

NOTATIONS AND PRELIMINARIES

Definitions and Lemmas concerninginformation

In this paper, we denote the vector of the sequenceof a (discrete-time) stochastic process ξ(k) ∈ R, (k =1, 2, . . .), as

ξ n = [ξ(n), ξ(n − 1), . . . , ξ(1)]T (1)

The entropy rate[13] of a stationary stochastic processX (k)

H (X ) =: limn→∞1

nH (X n) (2)

describes the per unit time information or uncertainty ofX , where H (X n) is the entropy of X n ; while the mutualinformation rate[13] between two stationary stochasticprocesses X and Y

I (X ; Y ) =: limn→∞1

nI (X n ; Y n) (3)

describes the time average information transmittedbetween processes X and Y , where I (X n ; Y n) is themutual information of X n and Y n . The notation ‘=:’means definition. The directed information [1,3] from thesequence X n to the sequence Y n is defined as

I (X n → Y n) =:n∑

k=1

I (X k ; y(k)|Y k−1) (4)

while the directed information rate is

�I (X → Y ) =: limn→∞

1

nI (X n → Y n) (5)

Some conclusions concerning entropy are stated asfollows.

Lemma 1: Let X , Y , Z be random vectors withappropriate (needless same) dimensions, and f (·) be adeterministic map. Then

H (x + f (y)|y) = H (x |y), (6)

H (x |f (y) + z , y) = H (x |z , y) (7)

where H (·|·) denotes the conditional entropy.[13]

Proof : See Eqn. Appendix A1.

Lemma 2[14]: Let F (z ) ∈ RH∞ be the transfer func-tion of a discrete time, single-input and single-output(SISO), invertible linear time invariant (LTI) systemwith variables taking values in continuous spaces,where RH∞ denotes the set of all stable, properand rational transfer functions.[15] The stochastic input

x(k) ∈ R (k = 1, 2, . . .) is stationary. Then the entropyrate of system output y(k) ∈ R is

H (y) = H (x) + 1

4π

∫ π

−π

ln |F (eiω)|2 dω (8)

Remark 3: The second term in the right hand ofEqn (8) reflects the variation of time average informa-tion of the signal after it transmitted through the systemF (z ), and was defined as the variety of system F (z ).[11]

Denote it as

V (F ) =:1

4π

∫ π

−π

ln |F (eiω)|2 dω (9)

Intrinsically, the system variety is caused by systemdynamics, or, memory. When V (F ) = 0, we say thatthe system F (z ) is entropy preserving.

Remark 4: The proof of Lemma 2[14] is based on thefact that for sequences of the input xn ∈ R

n and outputyn ∈ R

n with yn = Fnxn , where Fn is the invertiblelinear transformation matrix defined by system F (z ),H (yn) = H (xn) + ln det J , where J is the Jacobianmatrix of Fn . However, if the number of samples ofx(k) and y(k) is finite, then H (yn) = H (xn). In thiscase, the Eqn (8) is modified as H (y) = H (x), i.e. thesystem is always entropy preserving.

The system under discussion

In this paper we will discuss the information transmis-sion in the discrete time SISO LTI tracking systemshown in Fig. 1: where r(k), d(k), u(k), y(k) ∈ R

are the reference input, disturbance, control signal andoutput, respectively, k is the time index; C (z ) and P(z )

are proper rational transfer functions of controller andplant, respectively.

Assumptions 5:

(a) The reference input r and the disturbance d aremutual independent stationary random sequences.The system has zero initial condition (i.e., for k ≤ 0,the variables in system are zero).

(b) The closed-loop system is well-posed and internallystable. The well-posedness requires the transfer

Figure 1. Feedback LTI tracking system withdisturbance.

2008 Curtin University of Technology and John Wiley & Sons, Ltd. Asia-Pac. J. Chem. Eng. 2008; 3: 630–637DOI: 10.1002/apj

632 H. ZHANG AND Y. SUN Asia-Pacific Journal of Chemical Engineering

function 1 + L(z ), where L(z ) = P(z )C (z ), is one-to-one and onto, i.e. invertible.[15,16]

(c) The open-loop transfer function L(z ) has no pole-zero cancellation and no pole or zero on the unitcircle in complex plane, and can be represented as:

L(z ) =l0

p∏i=1

(z − zi )

q∏j=1

(z − pj )

(10)

where l0 �= 0 is the leading coefficient of the numeratorof L(z ) when the dominator of L(z ) is monic, and ischosen to stabilize the closed system; zi and pj are zerosand poles of L(z ), respectively.

The closed-loop transfer functions are S (z ) = [1 +L(z )]−1, T (z ) = 1 − S (z ). The response of S (z ) tothe disturbance in a finite time interval can be repre-sented as

yd (k) =k∑

i=1

sk−i d(i ) (11)

where sk−i ’s are response weighting parameters. Thesignal yd (k) and vector yk

d are linear functions of theinput sequence dk . Denote them as

yd (k) =: Sk dk , ykd =: Sk dk (12)

respectively, where

Sk = [s0 s1 · · · sk−1],

Sk =

Sk

0 Sk−10 0 Sk−2

...0 · · · 0 S1

(13)

are linear deterministic maps. It is seen in our systemthat Sk is an invertible transformation matrix. Wealso denote respectively the linear maps Tk and Tk

corresponding to the transfer function T (z ), Lk and Lk

corresponding to the open-loop transfer function L(z ),Pk , Pk corresponding to P(z ), and Uk , Uk correspondingto U (z ) = C (z )S (z ), in the same sense as Sk and Sk .For discrimination, we will denote the variables in open-loop system with the subscript “o”. For examples, uodenotes the control variable in open-loop system, whileu denotes that in the closed-loop system; r denotes thereference input in open- and closed-loop system becauseit is the same in both cases.

By using these notations, we can write

yo(k) = Lk rk + d(k), yko = Lk uk + dk (14)

for the open-loop system. For the closed-loop system,we have

y(k) = yr (k) + yd (k), yk = ykr + yk

d (15)

u(k) = ur (k) − ud (k), uk = ukr − uk

d (16)

where yr (k) =: Tk rk , ykr =: Tk rk , ur (k) =: Uk rk ,

ud (k) =: Uk dk , ukr =: Uk rk , uk

d =: Uk dk . We also have

y(k) = Pk uk + d(k), yk = Pk uk + dk (17)

for both open- and closed-loop systems.

Lemma 6:[17] Under the conditions stated in Assump-tion 5, the Bode integral of sensitive function S (z ) ofthe closed-loop stable discrete-time LTI system shownin Fig. 1 satisfies

∫ π

−π

ln |S (ejω)|dω = 2π

(m∑

i−1

ln |pui |

− ln |ζ + 1|)

(18)

where the pui ’s, i = 1, · · · , m , are unstable (i.e. outside

the unit disk in complex plane) poles of L(z ), andζ = limz→∞ L(z ).

Note that in the above equation, ζ = 0 if the open-loop transfer function L(z ) is strictly proper (p < qin Eqn (10)), and ζ = l0 if L(z ) is biproper (p = q inEqn (10)).

INFORMATION TRANSMISSION IN LTICONTROL SYSTEMS

The tracking system shown in Fig. 1 is similar toa channel with intersymbol interference (ISI),[3] to alarge degree. The reference input r , control signal u ,and system output y can be considered as the sourcemessage, encoded channel input, and channel output,respectively.

If the open-loop system is stable, the spectrum of theoutput is

�yo(ω) = |L(ejω)|2�r (ω) + �d (ω) (19)

where �r and �d are spectrums of r and d , respec-tively. For the closed-loop system,

�y(ω) = |T (ejω)|2�r(ω) + |S (ejω)|2�d (ω)

= |S (ejω)|2[|L(ejω)|2�r(ω)

+ �d (ω)] (20)



Hence, y can be considered as the response of systemS (z ) to stationary input y0. Then by Lemma 2 andLemma 6 we get the following conclusion.

Proposition 7: For the open-loop stable feedbacktracking system satisfies Assumption 5, the entropyrates of the outputs of open- and closed-loop systemshave relation

H (y) = H (yo) + V (S )

= H (yo) − ln |ζ + 1| (21)

where V (S ) =: 14π

∫ π

−πln |S (eiω)|2 dω is the variety of

system S (z ). If the open-loop transfer function L(z ) isstrictly proper (p < q in Eqn (10)), then

H (y) = H (yo) (22)

Remark 8: It can be seen from Lemma 6 andProposition 7 that for a stable and strictly proper L(z ),the feedback does not change the output uncertainty.For biproper L(z ), the output uncertainty is reducedby feedback if |ζ + 1| > 1. However, from Remark4 we know that if the number of samples of systemvariables is finite, then the system S (z ) is alwaysentropy preserving, and the feedback does not changethe output uncertainty even if L(z ) is biproper.

Information transmission from extraneousinputs to internal variables

In this section, we will investigate the informationtransmission between two pairs of extraneous andinternal variables, (r , y) and (d , u), respectively.

Proposition 9: Suppose the system shown in Fig. 1satisfies conditions stated in Assumptions 5, then,

I (r ; yo) = �I (r → yo) = H (yo) − H (d) (23)

and,

I (r ; y) = �I (r → y) = H (y) − H (d)

− V (S ) (24)

I (r ; y) = �I (r → y) = I (r ; yo)

= �I (r → yo) (25)

if the open-loop system L(z ) is stable

Proof: See A2.

We then consider the information transmissionbetween d and u .

Proposition 10: Suppose the system shown in Fig. 1satisfies conditions stated in Assumptions 5, then

�I (d → u) = I (d ; u) = H (u) − H (ur ) (26)

Proof: See A3.

Remark 11: Propositions 9 and 10 state that feed-back does not change the information transmission fromextraneous inputs to internal variables. This is slightlydifferent from the conclusion in the communicationchannel, which states that for continuous alphabet Gaus-sian channels with colored noise, the capacity (definedas the maximum mutual information between messageand channel output) is increased by feedback.[13] Fur-thermore, with similar analysis as in the proofs ofPropositions 9 and 10, it can be concluded that for thepairs of extraneous input variable and internal variable(such as (r , y), (d , y), (r , u), and (d , u)), mutual infor-mation and directed information are equivalent in bothcases of open- and closed-loop systems. This propertyis based on the fact that the feedback does not changethe statistic (in)dependence between the future inputsand the current (and previous) internal variables. Specif-ically, let e(k) = r(k) − y(k) be the tracking error inclosed-loop system, then e(z ) = S (z )[r(z ) − d(z )]. Wecan get

I (rn → en) = I (rn ; en) = H (en)

− H (Sndn) (27)

Then if the system is Gaussian,

�I (r → e) = I (r ; e)

= H (r − d) − H (d)

= 1

4π

∫ π

−π

ln

(1 + �r

�d

)dω (28)

Hence, the information transmission from r to e isdefined uniquely by the signal–noise ratio.

As stated in Ref. [1], the directed information is notsymmetry. Considering a ‘fictitious’ directed informa-tion from internal signal to extraneous input signal willthrow light on an interesting relation. Define the ficti-tious directed information (rate) from un to dn as:

I (un → dn) =:n∑

k=1

I (uk ; d(k)|dk−1), �I (u → d)

=: limn→∞

1

nI (un → dn) (29)



We have

I (un → dn)

=n∑

k=1

I ((uk−1, u(k)); d(k)|dk−1)

(a)=n∑

k=1

[I (uk−1; d(k)|dk−1)

+ I (u(k); d(k)|dk−1, uk−1)]

=n∑

k=1

[H (d(k)|dk−1) − H (d(k)|dk−1, uk−1)

+ I (u(k); d(k)|dk−1, uk−1)]

(b)=n∑

k=1

[H (d(k)|dk−1) − H (d(k)|dk−1, uk−1r )

+ I (u(k); d(k)|dk−1, uk−1)]

(c)=n∑

k=1

I (u(k); d(k)|dk−1, uk−1) (30)

where (a) is based on the chain rule of mutualinformation,[13] (b) is based on (7) and (c) on the factthat d and r are mutually independent. On the otherhand, with the chain rule of mutual information,

I (dn → un) =n∑

k=1

[I (dk−1; u(k)|uk−1)

+ I (d(k); u(k)|uk−1, dk−1)] (31)

where I (dk−1; u(k)|uk−1) = H (dk−1|uk−1)

− H (dk−1|uk−1, u(k)) ≥ 0 with equality if and only ifd(k) is white. Hence,

I (dn → un) ≥ I (un → dn), �I (d → u)

≥ �I (u → d) (32)

with equalities if and only if d(k) is white.

Information transmission between internalvariables

Only the pair of control variable and output is consid-ered in this section.

Proposition 12: Suppose the system shown in Fig. 1satisfies conditions stated in Assumptions 5. If the open-loop system with L(z ) is stable,

�I (uo → yo) = I (uo; yo) = H (yo) − H (d) (33)

In the closed-loop system,

�I (u → y) ≤ I (u; y) (34)

with equality holds if and only if d is white, where

�I (u → y) = H (y) − H (d)

+ �I (u → d), (35)

I (u; y) = H (y) − H (d)

+ �I (d → u) (36)

Proof: See A4.

Remark 13: Although in the closed-loop system,directed information and mutual information may beidentical, the feedback changes the information trans-mission between internal variables even if the distur-bance is white. This can be seen in the following rela-tion derived from Eqns (33),(35), and (21),

�I (u → y) − �I (uo → yo) = V (S )

+ �I (u → d) (37)

where the quantity on the right-hand side of the equalityis not identical to zero in general. Let us considera special case when d is white, C (z ) is invertible,and the system is Gaussian. With Eqns (26),(32), andLemma 2 we get �I (u → d) = �I (d → u) = 1

4π

∫ π

−π

ln(1 + �d�r

)dω, and hence

�I (u → y) − �I (uo → yo) = V (S )

+ 1

4π

∫ π

−π

ln(1 + �d

�r)dω (38)

If S (z ) is entropy preserving, the variation of infor-mation transmission caused by feedback is a constant.

CONCLUDING REMARKS

For pairs of system extraneous inputs and internalvariables (including system output), the directed infor-mation (rate) is always equal to the mutual infor-mation (rate); For the pair of internal variables,the former is smaller than or equal to the latter.And, the feedback changes the information transmis-sion between internal variables, while it makes noinfluence on information transmission from extrane-ous variables to internal variables. Our conclusionis slightly different from that in the communicationchannel.[13]

In designing of communication systems, one alwaysintends to design the probability distribution of the



channel input to maximize the information transmissionfrom channel input to output, to achieve the chan-nel capacity. However, in our tracking system, neithermaximizing information transmission from r to y normaximizing information transmission from u to y isa suitable choice to get good tracking performance.The key distinction between structures of tracking sys-tem and the communication channel is that, in a com-munication system a postfilter, the decoder, is usedbefore transmitted signal is received by user, whilewhereas in a tracking system there is no postfilter.From Proposition 9 (Eqn (24)) and Proposition 12(Eqn 35) we see that maximizing �I (r → y) or �I (u → y)

may make H (y) too large. This implies the outputmay contain more information or uncertainty than thereference signal. Therefore, maximizing �I (r → y) or�I (u → y) can not be used directly in tracking con-trol systems. Another intuitive choice is minimizingthe information transmission from reference to trackingerror. However, �I (r → e) is not a suitable performancefunction, too, because �I (r → e) is defined by the sig-nal–noise ratio (Remark 11) and is independent of sys-tem parameters. In our viewpoint, a rational choice is toadopt �I (r → y) as an auxiliary performance function,as discussed in[12.]

APPENDICES

A1. Proof of Lemma 1

Let ε = x + f (y) denote certain semi-open intervals

>ε0 ≤ ε < ε0 + dε, x0 ≤ x < x0 + dx (A1.1)

for random variables ε and x , respectively. For a fixedy = y0, random events ε0 ≤ ε < ε0 + dε and x0 ≤ x <x0 + dx are one-to-one. The probability

P(ε0 ≤ ε < ε0 + dε|y = y0) = p(ε|y)|dε| (A1.2)

equals the probability

P(x0 ≤ x < x0 + dx |y = y0) = p(x |y)|dx | (A1.3)

where |dx | denotes |dx1| · |dx2| · · · |dxn |, n is the dimen-sion of x . Then

p(ε|y) = p(x |y)|dx ||dε| = p(x |y)

|dx ||dεT| (A1.4)

For dxdεT = I , | dx

dεT | = |det[ dxdεT ]| = 1, we have

p(ε|y) = p(ε − f (y)|y) = p(x |y) (A1.5)

Moreover, p(ε, y) = p(x , y). Then, by the definitionof conditional entropy,[13] Eqn (6) is arrived at.

The proof of (7) is given as:

H (x |f (y) + z , y)

= H (x |y) − I (x ; f (y) + z |y)

= H (x |y) − H (f (y) + z |y) + H (f (y) + z |x , y)

= H (x |y) − H (z |y) + H (z |x , y)

= H (x |y) − I (x ; z |y))

= H (x |z , y) (A1.6)

where the third equality is based on (6). �

A2. Proof of Proposition 9

First, we consider the open-loop system (i.e. there is nofeedback in Fig. 1) with L(z ) stable. With Eqn (14) weget the mutual information between rn and yn

o as

I (rn ; yno ) = H (yn

o ) − H (yno |rn)

= H (yno ) − H (Lnrn + dn |rn)

= H (yno ) − H (dn) (A2.1)

where the third equality is based on (6) and the factthat d and r are mutually independent. Then the mutualinformation rate is

I (r ; yo) = H (yo) − H (d) (A2.2)

The directed information is

I (rn → yno )

=n∑

k=1

[H (yo(k)|yk−1o ) − H (yo(k)|yk−1

o , rk )]

=n∑

k=1

[H (yo(k)|yk−1o ) − H (Lk rk + d(k)|yk−1

o , rk )]

(d)=n∑

k=1

[H (yo(k)|yk−1o )

− H (d(k)|Lk−1rk−1 + dk−1, rk )]

(e)=n∑

k=1

[H (yo(k)|yk−1o ) − H (d(k)|dk−1] (A2.3)

where (d) is based on (6), (e) is based on (7), and thefact that d and r are independent. Then the directedinformation rate is

�I (r → yo) = H (yo) − H (d) (A2.4)



Second, let us consider the closed-loop system. Forthe sequences of rn and yn , we have

I (rn ; yn) = H (yn) − H (yn |rn)

= H (yn) − H (Tnrn + ynd |rn)

= H (yn) − H (ynd ) (A2.5)

where the third equality is based on Eqn (6) and thefact that d and r are independent. Then the mutualinformation rate between r and y is

I (r ; y) = H (y) − H (yd ) (A2.6)

With Lemma 2 and Eqn (12), we have

I (r ; y) = H (y) − H (d) − V (S ) (A2.7)

The directed information rate from r to y is

I (rn → yn)

=n∑

k=1

[H (y(k)|yk−1) − H (y(k)|yk−1, rk )]

=n∑

k=1

[H (y(k)|yk−1) − H (Tk rk

+ yd (k)|Tk−1rk−1 + yk−1d , rk ]

(f)=n∑

k=1

[H (y(k)|yk−1) − H (yd (k)|Tk−1rk−1

+ yk−1d , rk ]

(g)=n∑

k=1

[H (y(k)|yk−1) − H (yd (k)|yk−1d ) (A2.8)

where the equalities (f) are based on Eqn (6); (g) isbased on (7) and the fact that d and r are independent.By the property of the entropy rate[13] we have

�I (r → y) = H (y) − H (yd ) = H (y)

− H (d) − V (S ) (A2.9)

With Eqns (A2.2)–(A2.9) and Proposition 7, we getthe conclusions. �


We have

I (dn ; un) = H (un) − H (un |dn)

= H (un) − H (Undn − unr |dn)

(a)=H (un) − H (unr ) (A3.1)

and

I (dn → un)

=n∑

k=1

H (u(k)|uk−1) − H (u(k)|uk−1, dk )]

=n∑

k=1

H (u(k)|uk−1) − H (ur (k) − Uk dk |uk−1r

− Uk−1dk−1, dk )]

=n∑

k=1

H (u(k)|uk−1) − H (ur (k)|uk−1r )] (A3.2)

Then with the definition of entropy rate we get

�I (d → u) = I (d ; u) = H (u) − H (ur ) (A3.3)

�


We first consider the open-loop system with L(z ) stable(This implies P(z ) is stable). In this case, we have

I (uno ; yn

o ) = H (yno ) − H (yn

o |uno )

= H (yno ) − H (Pnun

o + dn |uno )

= H (yno ) − H (dn

o ) (A4.1)

with Eqn (6) and the fact that d and u are independentin open-loop system. And

I (uno → yn

o )

=n∑

k=1

[H (yo(k)|yk−1o ) − H (Pk uk

o + d(k)|yk−1o , uk

o )]

=n∑

k=1

[H (yo(k)|yk−1o ) − H (d(k)|Pk−1uk−1

o

+ dk−1, uko )]

=n∑

k=1

[H (yo(k)|yk−1o ) − H (d(k)|dk−1] (A4.2)

Then we consider the closed-loop system. WithEqns (18) and (6), it can be understood that the mutualinformation is

I (un ; yn) = H (yn) − H (yn |un)

= H (yn) − H (Pnun + dn |un)

= H (yn) − H (dn) + I (dn ; un) (A4.3)



The directed information in the closed-loop system is

I (un → yn)

=n∑

k=1

[H (y(k)|yk−1) − H (y(k)|yk−1, uk )]

=n∑

k=1

[H (y(k)|yk−1) − H (Pk uk + d(k)|yk−1, uk )]

=n∑

k=1

[H (y(k)|yk−1) − H (d(k)|Pk−1uk−1

+ dk−1, uk )]

=n∑

k=1

[H (y(k)|yk−1) − H (d(k)|dk−1, uk )]

=n∑

k=1

[H (y(k)|yk−1) − H (d(k)|dk−1)

+ I (d(k); uk |dk−1)]

= H (yn) − H (dn) + I (un → dn) (A4.4)

Then (33)–(36) are arrived at by using (26),(32),and (A4.1)–(A4.4). �

REFERENCES

[1] J.L. Massey. Causality, feedback and directed information.In Proceedings 1990 International Symposium on InformationTheory and its Applications , Waikiki, Hawaii, Nov. 1990;27–30.

[2] C.E. Shannon. Bell Syst. Tech. J., 1948; 27, 379–423,623–656.

[3] S.-H. Yang. The capacity of communication channelswith memory. PhD Dissertation, Cambridge, Massachusets,Harvard University, May 2004.

[4] N. Elia. IEEE Trans. Automat. Contr., 2004; 49(9),1477–1488.

[5] S. Tatikonda. Control under Communication Constraints.PhD Dissertation, Cambridge, Massachusets, MassachusettsInstitute of Technology, Sept. 2000.

[6] S. Engell. Kybernetes, 1984; 13, 73–77.[7] G.N. Saridis. IEEE Trans. Automat. Contr., 1988; 33,

713–721.[8] A.A. Stoorvogel, J.H. Van Schuppen. System identification

with information theoretic criteria. In Identification, Adapta-tion, Learning (Eds.: S. Bittanti, G. Picc), Springer: Berlin,1996; pp.289–338.

[9] L. Wang. J. Syst. Sci. Complexity, 2001; 14(1), 1–16.[10] H.L. Weidemann. Entropy analysis of feedback control

systems. Advances in Control Systems, Vol 7, Academic Press,New York: 1969 pp pp.225–255.

[11] H. Zhang, Y.-X. Sun. Bode integrals and laws of variety inlinear control systems. In Proceedings 2003 American ControlConference, Denver, Colorado 2003.

[12] H. Zhang, Y.-X. Sun. Information theoretic interpretations forH∞ entropy. In Proceedings 2005 IFAC World Congress ,Prague, 2005.

[13] S. Ihara. Information Theory for Continuous Systems, WorldScientific Publishing Co. Pte. Ltd.: Singapore, 1993.

[14] A. Papoulis. Probability, Random Variables, and StochasticProcesses, 3rd edn, McGraw-Hill, Inc.: New York, 1991.

[15] K. Zhou. Essential of Robust Control, Prentice-Hall: UpperSaddle River, NJ, 1998.

[16] J.C. Willems. The Analysis of Feedback Systems. The MITPress, Cambridge, Massachusets: 1971.

[17] B.-F. Wu, E.A. Johckheere. IEEE Trans. Automat. Contr.,1992; 37(11), 1797–1802.


Documents

On information transmission in linear feedback tracking systems