196 IZVESTIYA VUZ. RADIOFIZIi<A

A GENERALIZED STOCHASTIC APPROXIMATION PROBLEM

V. A. Brus in

Izvest iya VUZ. Radiofizika, Vol. 11, No. 3, pp. 353-367, 1968

UDC 621.391.192.5

Discrete and continuous versions of a generalized stochastic approxi- mation problem are formulated. It is shown that this problem can be solved by the Robbins-Murtto algorithm which converges in probabil- ity.

The problem of stochastic approximation was f i r s t formulated and solved by Robbins and Munro [1]. Having a r i sen in connection with r eg ress iona l analy- s is , the problem consis ts in set t ing up an i tera t ive process that converges in probabi l i ty to the root of the equation E w(~)=0, where w(x) is a random func-

tion of a rgument x, and E is the mathemat ica l ex- x

pectation of this random function for the stated value of the argument . Robbins and Munro found such a pro- cess and demonstra ted its convergence in probabi l i ty to the root, subject to ce r ta in res t r i c t ions on the ran - dom function w(x).

Paper [1] was the fo r e runne r of many studies of stochastic approximation. A fa i r ly complete survey of these papers is to be found in the journal Avtomatika i Telemekhanika, No. 4, 1966.

Two years ago, these papers came to the at tention of specia l i s t s working in the field of applied cyber - net ics . As was f i r s t pointed out by Tsypkin [2], the stochastic approximation problem proves to be closely connected with severa l p resen t -day problems of cy- berne t ics , e . g . , pa t tern recognit ion, dual control , etc. Tsypkin [2] also in terpre ted stochast ic approxi- mation as a problem of the (stochastic) s tabi l i ty of a cer ta in type of nonl inear nonautonomous stochast ic sampl ing control sys tem (see Fig. la). Subsequently, the Robbins-Munro problem was general ized to the case of continuous search which cor responds to the analogous continuous control sys tem (see Fig. 2a).

The p resen t paper gives a fur ther genera l iza t ion of the Robbins-Munro problem to problems associated with more complicated control sys tems . F igure lb shows the control sys tem corresponding to the d i s - cre te ve r s ion of the problem, and Fig. 2b is for the continuous vers ion . It has been proved that the Rob- b ins -Munro algori thm for the "c lass ica l" problem of stochastic approximation also solves the above gen- eral ized problem for the same res t r i c t ions on the r an - dom quanti ty ~.

The d isc re te case is considered in detail . For the continuous case, in view of the analogy, it suffices to formulate the problem and quote the bas ic resu l t s .

1. FORMULATION OF PROBLEM FOR DISCRETE CASE

Let us consider a nonl inear s tochast ic object with controlled input {Un}, uncontrol led s tochast ic s ignal (in}, and output {Vn} (n = - 1 , 0, 1, 2 , . . . ) .

The output s ignal {vn~ is related to the input va r i - ables via a difference equation

v~, = w,,_q (q > 0 ) , (1)

w,, = r + %, (2)

Tox , + Tlx, ,-1 + . . �9 + T~x . . . . "court +

+ ~ltt.-t + . �9 �9 + ~ u . . . . (3)

where {Xn}, {Wn} are "internal" variables in the ob-

ject.

The block d iagram of the object is given in Fig. lb . The state of the object (1) through (3) at ins tant k is given (see [3]) by the s -d imens iona l vector (x k, Xk- 1,

. . . . Xk-s+l). We wil l a ssume that at ins tant n = 0 the object is in

the state M ~ given by

(Xo o, xo_t . . . . . .~_~+,) (4a) and that

u. = 0 (n ~< 0) . (5)

Moreover, the following proper t ies of the object will be postulated.

NONLINEARITY OF go (x)

I) There is a | such that

~(~) = 0 , (6a)

and when x # | we have

~(x) (x -- O) ~ 0. (6b)

2) For all x there is an R < co such that

I v(x)! ~< R. (6c)

3) For a r b i t r a r y 6 > 0 there is a A(5) > 0 such that

i ? (x) l ~ A, (6d)

when Ix - @1 > 6.

LINEAR PART OF OBJECT

1) The roots of the polynomial

T~(z) ~ To zs + T l z s- l -]- . . . + Ts (7a)

l ie inside the unit c i rc le . s

2) ro + o, ~ ~ + o. (7b) 0

RANDOM PROCESS {In}

The random process {in} = in(A) is defined in the usual way as a function of two var iab les , namely the in tegers n and )% where ~ is an e lement of a set A on which is defined a (y-algebra and the probabi l i ty mea- sure P(~), also called the probabi l i ty space (see [4]).

RADIOPHYSICS AND QUANTUM ELECTRONICS 197

For a fixed ~ in A we have a function of the va r i - able n. This const i tutes a rea l iza t ion of the random process .

According to (1), (2) the output process v n, and w n also, are random processes on the same probabi l i ty space A. (Thus s t r i c t ly speaking we should wri te Vn(k ) and Wn(A)). Let A0 be a subse t of A, and let E

h0

denote the conditional mathemat ica l expectation on A0, i . e .

Moreover , ~k will denote the (k + 1) -d imensional vec- tor (~0 . . . . . ~k), and ~ s (k)wil 1 denote the mathemat i -

cal expectation of s(X) on the set A(~k) defined by the conditions ~i(~) = ~i (i = 0, 1 . . . . . k).

The r e s t r i c t i on on the random process {~k} can now be formulated as

E ~ , ~ = 0 ( ] = l + q , 2q-q . . . . , n),

n - ]

E ~ < o ~ < ~ ( n = o , ~, 2, ) (8)

Now the given object is defined by Eqs. (1)-(5), (6a)-(6d), (Ta), (7b), and (8). At each instant of t ime n, the value of the output w n is known and the input u n can be p resc r ibed . The general ized s tochast ic approx- imat ion problem consis ts in finding an a lgor i thm f

a,, = f ( v ~ - l , v n - 2 . . . . ; u n - 1 , u , - 2 , �9 . . ) , (9)

that is independent of the values of the object p a r a m - e ters and such that the process {Un} converges to the root | of the equation

~(x) = o . (lO)

A more p rec i se formulat ion of the problem is as fol- lows: to find an algori thm (9) such that the random process {Xn}, generated by {Un} in accordance with (2) and (3), converges in probabi l i ty [4] to the root of equation (10), i . e . ,

(p) llm [Xn} = O. (11)

tg

In w167 3 below it will be shown that this problem is solved by the Robbins-Munro algori thm [1]:

un+l = Un - - 7nvn (n = O, 1 .... ), (12a)

where {Yn} is a s i gn - inva r i an t sequence (or contains only a finite number of changes in sign) such that

co ca

E T~ = co, E ~ ] < co . (13a) rz=O n=O

2. SOLUTION OF THE GENERALIZED STOCHASTIC

APPROXIMATION PROBLEM (DISCRETE VERSION)

Consider the recursive formula

= - (E r,) (12b)

where Yn > O. Condition (13a) will be satisfied when n -> 0, and it will be assumed that

Tn = 0 (when n ~ O ) . (13b)

In v i r tue of (1) and (3) the algori thm (12a) generates the random process {Xn(k)} where k belongs to A. The block d iagram of sys tem (1)-(3), (12b) is shown in Fig. lb . We have the following theorem.

Theorem 1. The random process [Xn(k)} generated by the algori thm (12b), (13a) converges in probabil i ty to the value |

Before proving the theorem we will set down some resu l t s f rom the theory of l inear difference equations and prove some l emmas .

It follows from (1)-(4a), (12b), and (13b) that

To(x~ - - x,~-0 +

-~- T l (xn- -1 - - x n - 2 ) q . . . . - k T + ( x . . . . - - x . . . . 1) =

=--P(~oY,-I-q-P~lY, 2 q q - . . . + % ~ y n - s - q I) ; (14a)

y. = ~.~,. = 7,,(,e(z.) -~ ~ . ) ,

x(0) = x0, x_ , = x ~ - l . . . . . x_~+t = x ~ (4b)

where for s impl ic i ty of notation the p a r a m e t e r X in xi(k ) has been omitted and the abbreviat ion ,o = sgn x

x ( S " ' / 2 r i ) i s used.

F rom (14a) it follows [3] that

~l--q--1

x n = - - p ~ H ( n - - m)y m -4- C o ( N o ) q- f ( t ~ , N o ) , , (15a) r n = 0

where H(n) is the inverse z - t r a n s f o r m of

l Y / ( z ) = "%z s - k "~tz ~ - 1 + �9 . . -}- %, (i6a) z q ( z - - 1) (Toz" + T ~ z ~ - ' + . . . + T D '

and the function C0(M0), f ( n , Mo) sat isfy the re la t ion

If(n, No) I< C,(M.)!'~[,* , (17a)

where ~ is the root of g rea tes t magnitude of the poly- nomial Ts(z), and

0

c0-,0, c , - , 0 (when Z = HNoll- ,0) . (17b) i = - - s + l

Let Q be an in teger g rea t e r than I + q and N* = s + + Q. We define the random process fin(k) (X E A, n = = . . . , - 1 , 0 , 1 , 2 . . . . ) such that

~o(x~ - - x,._,) +

: -- P(~oY~-I-q+ :lY~-2-q+ . . . + ' : ~ Y , ~ - ~ - t - q ) , (14b)

' ~ ( ~ ( x D + ~)

Yk = ~k(M~, Q)

0

where

( if k ~ N*)

( i f 0 < k ~ < , v * ) ,

(if k ~ 0)

(18)

198 I Z V E S T I Y A VUZ. RADIOFIZIKA

and the funct ion ~2 is such that

.q (~) = 0

(k = N* - - s - - q - - 1 . . . . . N* - - I, N*) ; (20)

XO = ~ ' - -1 = �9 �9 �9 = X - s + l ~ 0 ,

X N * - - i = X - - i ' - O ( i = 0 , 1 . . . . , s - - l ) . (21)

Condi t ion (20) m e a n s that a f t e r N* s t eps the c o n t r o l func t ion Yk = ~2k d r i v e s s y s t e m (14b) f r o m the z e r o in i t i a l s t a t e M0 = (0 . . . . . 0) into the s t a t e I~IN, which ,

z o ~ ( x ] ]r v,

a

b

Fig. 1

a p a r t f r o m a phase shi f t | c o r r e s p o n d s to s t a t e M 0 of the in i t i a l s y s t e m . This s t a t e is i d e n t i c a l l y z e r o on the fo l lowing s e q u e n c e of s t eps s + q + 1.

Such a con t ro l a lways e x i s t s (with c o r r e s p o n d i n g Q) if the s y s t e m is " c o m p l e t e l y c o n t r o l l a b l e " in the s e n s e of K a l m a n [5]. Condi t ions (7b) g u a r a n t e e th is p r o p e r t y .

It can be shown that when n > N*

X n ( X ) = X a t _ N , ( t , ) - - (~) . (22)

Thus , when n > N* the r e a l i z a t i o n s of the s t o c h a s t i c p r o c e s s Rn(X) s a t i s fy the s y s t e m of equa t ions (14b) and (18)-(20) wi th the in i t ia l cond i t ions (21) at the ins tan t n --: N*. A c c o r d i n g to the un iquenes s t h e o r e m t h e s e condi t ions un ique ly d e t e r m i n e Xn(it) fo r n > N* and fo r X E A .

In v i r t u e of (19), the change of v a r i a b l e s Yn = = Yn-N*, Xn = Xn-N* - | arid phase sh i f t ~ = n - N* changes s y s t e m (14b), (18) fo r fi > N* into s y s t e m (14a) fo r n > 0, and the in i t i a l cond i t ions (21) into (4b). In addi t ion, condi t ion (20) is t r a n s f o r m e d into cond i t ion ~ = 0 (H ~ 0). T h e r e f o r e condi t ion (22) wi l l be s a t i s - f ied when n > N*. Hence we have e s t a b l i s h e d the fo l - lowing l e m m a :

L e m m a 1. The c o n v e r g e n c e of the p r o c e s s xn(7~) to | is equ iva l en t to the c o n v e r g e n c e of the p r o c e s s XnQ') def ined by (14b) and (18)-(21) to z e r o .

Now the p roo f of the t h e o r e m r e q u i r e s a d e m o n - s t r a t i o n of the f ac t that

- (p) lira x n = O . ( 2 3 )

We wi l l p r e d e t e r m i n e the r a n d o m p r o c e s s ~n(X) in the i n t e r v a l 0 -< n -<- N* by se t t i ng ~n = 0. In v i r t u e of (8a) and (19), fo r n -> 0 we have

E { ~ - - 0 ( / = l + q , 2 + q . . . . . n ) ,

h

(~,-/ = (~o . . . . . T . - i ) ; n = 1, 2 . . . . ) . ( 2 4 )

Analogous to (15a) - (17b) , fo r the p r o c e s s ~n(X) we have when n > N > 0

n--q--I ~,('L) = - - p ~ H ( n - - r n ) ~m +

m:N--q--I

+ Co(MN) + f ( n -- N, ~I~,), (15b)

w h e r e lVI N is the s t a t e v e c t o r (RN, RN_I . . . . . XN-s+l) and C O and f a r e as in (15a).

To s i m p l i f y the no ta t ion we wi l l t e m p o r a r i l y o m i t the b a r s o v e r v a r i a b l e s x n, Yn, ~n, e t c . , r e m e m b e r - ing that f r o m now on we a r e dea l i ng wi th the a u x i l i a r y p r o c e s s .

Let us write W(z) given in (16a) in the form

koz-q k lg-q W(z) - ~.~ z-- { + K(z)' K(z) = - - , (16b)

Z - - Z t i=l

w h e r e z i a r e the po les of W(z), i . e . , the roo t s of Ts(Z ), w h o s e modu l i a r e l e s s than one, and k 0 = = ~ ~.~/~ T v It is known [3] tha t t h e r e is a cons t an t C 2 such that

SuplK(z)i ~ c~, 1 i" I= " [zl--i ~ - I K(e i'~) d m < C~ (25)

We i n t r o d u c e the notation

YN, r ( n ) = { y~ ( : i f N - < n ~ < T + N ) , (26) 0 i i . f ' n > T . q - N or n < N )

w h e r e YN, T(ejc~ is the F o u r i e r t r a n s f o r m of YN, T(n). F ina l l y we i n t r o d u c e the no ta t ion

N+T

~r(~N-~-~) = E ~ x~ y.0), (27) ~N--q--1 n:N

T

P~ = E ~ x.(x) y.(x) (28) A n=O

L e m m a 2. F o r a r b i t r a r y N -> 0 we have the inequa l - i t i e s

"pT(~N--q--I) < - - - -

~N-q--I n=N

N+T + [E (

~N--q--I n=N

N - - I

r)I=N--Q--I

RADIOPHYSICS AND QUANTUM ELECTRONICS 199

N+T

+ 2 2. rVCo(~,~ + ~) + n=N

N+T

-t-(2. ,~)1~1'~ [~//-~ Clt2(t~f 1 -{- o.,)l] ~ X

N - I

• (1 + l ~'I)-!;~+ #,',G N~.]+ L(N, N*), (29a) m=N--q--I

where L(N, N*) = 0 when N > N* + q + 1.

In virtue of (15b), (16a), and (26), when N + T -> _ > n _ > N _ > 0

n--q--1 + co ~y x~= --t1% 1 ~ ~,r(m)--p ~ k(n--m) y~,,r(m) +

m ~ -- co fft ~ --oo

-,~ Co(M~) + f (n - - N, m,) --

N--I N--I

N-q--I N--q--I

where k(n) is the inverse t rans form of K(z). Then + c o

;(~-~-~)=-I~01E E • ~N--q--I n=- -oz

n--q--I

X 2. Y ~ v , r ( m ) Y N , r ( n ) -

+oo +(.~

--? E E 2 k ( n - - m ) Y~,r(m)Ym~(n)+ ~ N--q--I --co --oo

N+T N+T

+Co(MN) E ~ Y. + E 2. f ( n - - N ) y . - - ~N--q--i n~-N {N--q--[ n=N

N--I N+T

-!~oI g X y~ } 2 y . - ~N--q--I re=N--q--1 n=N

N+T N--I

--o E ~ E k(n--m)y,nYn. (29b) { N - q - I rl=N m=N q--!

Making use of the P a r s e v a l theorem for latt ice functions [3] and taking account of (25), (26), (6e), and (8), for N > N* + q + 1 we obtain

+ c o + 2 o

~ N - - l l - I - -oo --co

r .<< ~ ] K (e j~ I E I ,~, ~-(e ~~ t 2 d~ < _~ ~N-q-.1

N+T

.< c~ E tV,v,r(e'~')l~a ''o= G ~ E x 2"2 gN--q--! n=N ~N--q ' l

--x

N+T

# < 2c~(~ ~ = ,~) Z r~ (30) X n ' " N

Use of the additional conditions (7a) and (24) y ie lds for N > N * + q + !

N+T N+T

t E Z ~ ( . - N ) y . I - < ( Z # ( ~ - m ~ ~N--q--~ n=N n=N

N+T co

• E 2. y~)',~< C'!"(MN)(Z { ,V--q- -1 m=N k=O

~ I'~) '/~ x

N+T 2~1/2 ~ - X • E Z y,) <V~ q'~(MN)

{N--q--| n=N

N+T

x (1 - - I z l ) - , ; ~ (R ~ + ~,~),,2( N . p l ';~ (31a) \ g,,--t /g] N

N+T N+T

E (y,.) < [E r t ~ N n~N

N--I N+T

E 2. 2., ~,~yo = m-N--q- - I n=N ~'N--q--I

N--I N+T

= E [ 5:y (E 2 . , ~ E x gA,'--q--I N--q--! ~N--I n=N ~N-q--!

N--I N+T

t2/y.)] j-<- E • m=N--q--I {N--I N ~N--q-I

N--I N+T

N--q--i {N--q--| N

N--I N+ T

r e = N - - q - 1 ~N--q--I N

N+T N ~ 1

E 5_', ~ k(,~.-,,,)y,,y,,, ~,V__d_l n~N ot=N~q.-I

N : T ,',,' - - 1

~c 2. ~ i k ( ,~ - 00 :',,'r,. x II N t?l = Nr--q ]

• ] g (%,,+~0,)(%~+~,,,'1 \ ' n - A' " N - q - 1

A ' - I N~T

x ~ I~(n- -m) l'r,,-~,~l E ~" '~ i < R ~'~? • m = N - - q l {N--q--i n='a/

V - I

\~ [ k(n .... m) "(,,7,,, <: RzC~ x x ..d ,n=, .V--q I

N - [ N T 5 3, 9 ~2

• -. :,,,(E~';,) �9 (ze) ;U N- q - 1 n.:.N

Applicat ion of the Parseva! theorem to the f i rs t term in (29b) y ie lds

+~J t~-q--I

E ~ E Y~,. Am) y,~,/.) = ~N-q--I n ~ - - ~ ra=--r

E i e - ' ; ; ~ - I ' F : v 7 ( d ~ ) l 2 d t u ~

:N-q-- . I --~

f; (e-';i"' -" ei''') (ei'~' " 1)-'t E I gtr r x 1

200 I Z V E S T I Y A VUZ. RADIOFIZIKA

x (d ~ l ~ d,,, = Yu, r +

i. co it

-I- E ~, ~ Yx, r ( m ) y, , , , r (n). (33a) ~N- q--i /~=-c~ ; t ; ~ - - c r

, - q l ~ S i n c e t h e r e i s a contour C~ > C 2 f o r w h i c h , e --

e / ' i i e ~ -- l I -~ f or ~ ~: [ - - ~, ~], it f o l l o w s w h e n N > N * + q + 1 that

N . { - t"

: , , , ,~ <: #~ - < 4 E E y~ < t t = N ~N--q--I

At-} , Y

3 '~) "1 -.< 2(C. , . - - C,.,) ( R ' -: " .~ "z,,, . (33b) N

It i s e a s i l y v e r i f i e d that

+co ~_~ 1 ,V.;.T ' YN, r(m) \' (n) >~ ( ~ y . t ~ (33c)

r n - -- oo ill ~ --~

Sub s t i tu t ion of (33b) and (33c) into (33a) , and s u b s t i - tut ion of i n e q u a l i t i e s ( 3 0 ) - ( 3 3 a ) into (29b) l e a d s to

a

Fig . 2

(29a) f or N > N* + q + 1. It i s e a s i l y s h o w n that the r ight s i d e of (29a) for the c a s e N -< N + q + 1 ca n be obta ined by addi t ion of the c o n s t a n t L > 0.

L e m m a 3. P o s i t i v e c o n s t a n t s C a, C4 e x i s t s u c h that f or a r b i t r a r y N -> 0, T > 0 w e h a v e

N 'i, T

0 < ~ 7,, ~ x.'?(x.) < Cs, (34) n ~= N ~,Y - q - - i

N ; T

E c,. ;,V--q--i :t g

F i r s t w e p r o v e that

N i T

:,,.(,~,,,-,,-!) = E ,',, E /1 " IV %

~ N - q - I

(35)

x , / f ( . r , , )+A(N, N*), (36)

w h e r e A(N, N*) = 0 w h e n N > N*. H e n c e , w h e n N > N*

N-I- T N ~- Y

E E x.y~ = E E ~,:,, ,, ( ,f ,, + ~,,) -= ~N--q--1 ,~=N n-,,}q "~,V-q [

N + T

: :s,-,,,, { E [ E u,,,?,, + n = N ~ N - q - I. -~,'~ q 1

,VIT Ni -T

= E ' r , , E E x,,% = E ' , ' . E .~,,~.(,~,,). (37a) n , N ~,V ,1-1 ~ n - , l - ! ~I=N ~A r q--1

The r e l a t i o n E x,~, = 0 u s e d h e r e i s a e o n s e - ~n--q--I

q u e n c e of the i n d e p e n d e n c e of x n and ~n-j (J = I, 2, . . . . q + 1) and c o n d i t i o n (8) . S i m i l a r l y , w h e n 0 -< N -< -< N* u s e of (18) l e a d s to

N , - T N + T N*- 1

~N- -q - - ] n ~ N n = N * ~,V q.-1 n- ,V

S u b s t i t u t i o n of (36) into (29a) y i e l d s (C~ ~- 2R 2 + 2a 2)

N + T

0 ~ 7. E x,,?,,<A(N, .u FL(N, N* ) - - N ~ N - - q - 1

--XIkol E 2 ~ n - q - 1

N + T

Co(M.) + n = N

N 1

+ r >5 E nz = N - q-- I ..~N_q l

N I - T

NI-T \n "f2 I1:

I I = N

N

N - 1 N + T

+ :~c~ =\' ~,'m I ( =' Y. , "2 l

N q--I n = N

H e n c e , in v i e w of (18) and (21) w e obta in

(38)

T co

, : ,' - + E < 21kol A n = 0 0

0

E N4- I" N + T ' \ 7

(39a)

N

+ 2 E " .7 < C~. (39b) !s

The relation (34) then follows from (35), (36), (37a, b),

(38) , and (39a, b) .

L e m m a 4 . F o r a r b i t r a r y N -> 0

~Ix,, i--x,~]-~O ( a s ' n ....... ) . (40) {g

When n > N + q + 1 it follows from (15b) and (16b) that

n- - q -- 1 n - q -- I

IV q- - ! N- . q - |

R A D I O P H Y S I C S AND QUANTUM ELECTRONICS

+ Co(MN) + f ( , . - ~, ~ ) ,

and hence

E !x~+,-x,,[~l~ol ~ I y . - ,~ l+ ~,v ~N

n -q

+ %~ !/f in + 1 - m) - - k(,z - - ,'.Oi E l Y,,, i f- ,n = q - 1 ~ N

+ I f ( n : ~ - - u ) - / ( n - ! v ) l .

(41)

Since E l.v,,, ! ~ ( El>' , , , /} ';~ -'~ 1/< r,,, ~" 0 a s m - ~ no, ~N A

it fol lows f r o m the p r o p e r t i e s of k(n), which is the in - t l - -q- - l

X; l te(n - - v e r s e t r a n s f o r m of K(z) in (16b), that In = / ' q - q - 1

- - m ) l ~ l y . , ] " 0 a s n ~ co. This c o m p l e t e s the proof

of (40). Let As , k(n) denote the s u b s e t of A def ined by

Ix,, (?,) I <: ~,

' x,,_~(~.)l :.C ~ . . . . . I x,,_~+~[).) I ~< ~ . (42)

L e m m a 5. F o r a r b i t r a r i l y s m a l l a > 0 and 6 > 0 the re is an N = N(Gd) such that

mcs.~: ~ ( N ) ~ . I - - a (k ::=0, t . . . . . s ) , (43)

w h e r e N can be t aken l a r g e r than any g iven n a t u r a l n u m b e r ~

F i r s t we e s t a b l i s h the va l id i ty of (43) for k = 0. The oo

s e r i e s .:. 1,, ~ xd?,, c o n v e r g e s in v i r t u e of (34). Thus ,

in view of (18) t he r e ex i s t s a s u b s e q u e n c e of i n t e g e r s {ni} such that

a',, i F,q -> 0 ( a s i -> o~) (44 ) A

We wil l denote the above s u b s e q u e n e e of i n t e g e r s by F. In view of (6b), (6d), and (19) we have

E .r A4, 0(lz) A -Q, 0 (n )

> i" x,,~.dP > a ; [ I rues \~,0(n)].

It fol lows f r o m (44) that

mes . , : ,0 f f0-* 1 ( a s lz -~ -, , n 6 F ) . (45)

Now we wi l l d e m o n s t r a t e the va l id i ty of (43) for k = s f r o m which fol lows i ts va l id i ty for a r b i t r a r y k -< s.

We wi l l c o n s i d e r the fo l lowing s u b s e t s of A. The s e t Uh(n ) of X E h is def ined such that

[ x . . - ~0 . ) - -&O, ) l < '~. (46)

and the se t US, s(n) is def ined by

A a (n - - s ) ,~ U,, ( n - - s ) :~. . . . . ~ Ua ( n - - I ) . (47a) , 0

a. ~ s

201

It is c l e a r that U6, s(n) C AS, s(n). F r o m w e l l - k n o w n t h e o r e m s in p robab i l i t y theory [4] it fol lows that

rues A~, ~(n) > rues U,.~(n) := rues AA 0 (n - s) :~ $

~.. ]![ rues U2_ ( n - 1). (47b) ] = 1 .r

Let ~ > 0 be a su f f i c i en t ly s m a l l n u m b e r . A c c o r d i n g to (45) a va lue N0 E F can be found such that

rues A,;, o(f f ,) -> t -- 7 . (48)

Making use of L e m m a 4 and the Chebyshev inequa l i ty we can p rove the ex i s t ence of a va lue Nj such that

rues U~ (n - - j) > I - - ~ -z

( n ~ > N i ; i .... 1, 2 , . , . , s). (49)

T h e r e is no loss in g e n e r a l i @ in a s s u m i n ~ Nj_< N0 (J = = 1 ,2 . . . . . s). Now we choose ~ such that (1--~) x x (I--7;, I ) s ~ ( 1 -Q , and se t N(e,5) = N0, w h e n c e ~ i n v iew of (47), i t fol lows that r ue s A<~(,V) .'~ (1 - - a) x x(1 - - ~-J)~ > (l - - ~) as was to be proved . S ince (43) is s a t i s f i ed for al l su f f i c ien t ly l a r g e N E F, the va lue of N m a y be taken to be a r b i t r a r i l y l a rge .

P roo f of T h e o r e m 1. We r e q u i r e to p rove (23), i. e . , [4] for a r b i t r a r y e > 0, 6 > 0 we wish to find a N(e ,5) > 0 such that for n > N

rues &(n) . (1 - - a), (50a)

w h e r e A6(n ) is the se t h E A def ined by the condi t ion

[.v:, i 5. (50b)

Let ~" and ~ be a r b i t r a r i l y s m a l l pos i t ive n u m b e r s . A c c o r d i n g to L e m m a 5 we can choose N('g,6) such that

, . es ~_ (~v) ~ (1 - 7 ) , (51) % s

M o r e o v e r , N can be chosen a r b i t r a r i l y l a rge and in- deed such that N > N* and

co _ N-I ~ ~ (52) y, ~. ~ c~; (N > ,u*)

/ V - - q . q

where C6, C 7 a r e cons t an t s . Le t ~ b e an a r b i t r a r y point in the se t A~ (N). We

wi l l c o n s i d e r inequa l i ty (29a) of L e m m a 2 for N = 17

and ~_ ~ ~~ (7). A c c o r d i n g to (42) we have N ~ q - - I N ~ q - - I

~, ] xi(~,) I .~< s~, and then in v i e w of (17b) we can ~ - - s + l

choose the c o n s t a n t Css -1 such that

I Co(M-) [2 < Cs'~, I CI(M.a)? 4 Cs~. (53) N

Then, f r o m (29a), (35), (36), and (38) it fol lows that

A r + T _ _

tkol E (~Y~ ~" ~ : :~ 'c , , (c~+V~c; )~+ 2

:V- -q- - I

202 IZ VESTIYA VUZ. RADIOFIZIKA

+ Gc:,Co~7 § I V ~ rc~ (~ - 1 7 b - "-~+

+ RaC~C~] Cr,~. (54)

Now cons ider re la t ion (41)o Squaring both sides and

applying the operator F~ , we obtain

N--,7--1

~r+r E ,,2_ <61ko l -~ E ( E Y . ) * +

N - - q - t N - - q - - I

- o o co

+ 6 ~ k~(n) ~\~ E Y~. + 61 C0(M-)~ l ~ + N--q-- I ~ N--q--I

- i-61G(m-)? + 6 1 k o 1 ' E •

N--q-I

Thus, in v i ewof (25 ) , (51), (52), (53), and (54)for a rb i t r a ry T > 0 we have

~ N " T

N. q--!

' 12C/~+ 12qlk o I ~ C,~Cs~ 7:--_ C~oYo, (55)

where C10 is independent of N, 5, and T. The set A~(N + T,%) is such that A E A and

M4 T a l -q - - I N- -q- l

According to the Chebyshev inequality, it follows f rom (55) that

rues k~(N § T, ~.) - (1 �9 C~,j~,-2). (56b)

With (43), (50b), (56a), and the well-known proper t ies of probabil i ty measure [4] we obtain

,.e~ ~,. (N + r ) (I -- C~o~,'~ ---~) (1 - -7) .

By choosing ~ and "6 so smal l that

(1. c f i 2 ~)(I ~) : -~(1--~). we obtain (50a), thus complet ing the proof of the the- orem.

3. CONTINUOUS VERSION OF THE GENERALIZED STOCHASTIC APPROXIMATION PROBLEM

Let us de te rmine the continuous analogs of (1)-(3), (4a), (45), (5), (6a)-(6d), (7a), (75), and (8).

Let u(t) be the controlled input signal , ~(t) be the uncontrol led stochastic input, and v(t) be the ou tpu t signal. Let x(t), w(t) denote the continuous analogs of Xn, w n in the d i sc re te case. The governing equations for the object are

~;(t) .... w ( t - q ) , (57) ~,(t) = .:(x(t)) + ~(t), (58)

Tox< ,~) + T~x(~-'~ + . . . + T~ -

= % u (~ ', ~ ,u ( ~ - b + . . . + % u . ( 5 9 )

The state M t of the object at ins tant t is descr ibed by the s -d imens iona l vector (x(t), x(l)(t) . . . . . x(S-0(t)). The nonl inear function q0(x) is s t i l l subject to condi- tions (6a)-(6d).

F u r t he r we will assume that the following condi- t ions s i m i l a r to (7a), (7b) are imposed on the non- l inear par t of the object (59):

1) The zeros Xi(i = 1, 2 . . . . . s) of the polynomial Tops + Tip s-1 + . . . + T s are in the lef t-hand half- plane

t?ek~ ~ 0 (i - 1, 2 , . . . , s). (60)

2) The condition of complete control labi l i ty [5] is sat isf ied. As shown in [6J, this r equ i res that

%~.i~" ~-' -.~}~--.~ ~ + . �9 -I- -~s ~- 0 ( i = l , 2, . . . , s ) . ( 6 1 )

By analogy with the d i sc re te ve rs ion we define the c lass of random processes }(t, ~) (X ~ A) [4]. We wilt assume that there is a v > 0 such that for a r b i t r a ry t > v and an a r b i t r a r i l y rea l iza t ion ~*(t) we have

~(t, ).) = 0 , (62) ,:*(t-9

where E is the conditional averaging operator on ~*(f--~)

the set A[~*(t - u)] defined by the condition

} (s, k) = !* (s) (.when 0 -~ s ~< (t -- v)) (63)

Moreover, for all t >- 0 the following inequali ty holds:

}2(t, i,) .<.: 32 < c~ , (64) A

The stochast ic approximation problem for the con- tinnous vers ion is formulated as follows: to cons t ruc t the algori thm

~(t) =d[u(s), v(s); 0 -< s < t] ,

which (a) is independent of the p a r a m e t e r s of the ob- ject, and (b) is such that the random process x(t, X) (t > 0, X E A) generated by this algori thm in accordance with (57)-(59) converges in probabi l i ty to |

For this problem there is a resu l t s i m i l a r to The- orem 1:

Theorem 2. The algori thm

where

co c~

~(t} > 0, S ~,' (t) dt = ~ , S y(t) dt < 0% {~ + ql >i ~, 0 0

and subject to conditions (6)-(64) solves the stochast ic approximation problem.

The proof of this theorem is a lmost ident ical with that of Theorem 1 with the lat t ice functions replaced

RADIOPHYSICS AND QUANTUM ELECTRONICS

by continuous functions and the summat ion signs r e - placed by in tegra l s , e tc .*

*The formula t ion of Lemmas 4, 5 in w is some- what different:

Lemma 4a. For a r b i t r a r y T > 0 and t o > 0

E F,~:(t+ r}--x(t)l~.0 (as t - -~ ) . (t~--,~)

Lemma 5a. Let the set As(t ) A be defined by

Then for a r b i t r a r i l y smal l e > 0, 5 > 0 a suff ici- ently large value of T(e,6) > 0 can be found such that mesAs (T ) -> (i - e).

The proof is based on the convergence of the inte- o~

gral ,f •( t )x( t ) ,~, (t) d~ a n d Lemma 4a, 0

203

REFERENCES

I. Robbins and Monro, Ann. Math. Star., 22, 400, 1951.

2. Ya. Z. Tsypkin, Avtomatika i telemekhanika, 27, no. i, 23, 1966.

3. Ya. Z. Tsypkin, Theory of Linear Sampling Systems [in Russian], Fizmatgiz, Moscow, 1963.

4. J. L. Doob, Stochastic Processes [Russian translation], IL, Moscow, 1956.

5. R. E. Kalman, Proceedings of International Congress of IFAC, izd. AN SSSR, Moscow, 1961.

6. V. M. l~opov, Avtomatika i telemekhanika, 24, no. i, 3, 1963.

27 Apr i l 1967 Scientific Research Inst i tute for Applied Mathematics and Cyber- net ics , Gor 'k i i Univers i ty