Vol.20 No.4 JOURNAL OF ELECTRONICS July 2003

C O L O R E D N O I S E P R E D I C T I O N B A S E D O N N E U R A L N E T W O R K 1

Gao Fei (Dept. of Electron. Eng. Info. and Tech., Beijing Institute of Technology, Bei fing 100081)

Zhang Xiaohui (Center for Space Science and Applied Research, Chinese Academy of Sciences, Belting 100080)

A b s t r a c t A method for predicting colored noise by introducing prediction of nonlinear time

series is presented. By adopting three kinds of neural networks prediction models, the colored noise

prediction is studied through changing the filter bandwidth for stochastic noise and the sampling

rate for colored noise. The results show tha t colored noise can be predicted. The prediction error

decreases with the increasing of the sampling rate or the narrowing of the filter bandwidth. If the

parameters are selected properly, the prediction precision can meet the requirement of engineering

implementation. The results offer a new reference way for increasing the ability for detecting weak

signal in signal processing system.

K e y w o r d s Nonlinear prediction; Colored noise; Time series; Neural network

I. I n t r o d u c t i o n

In recent years, the theories and methods on prediction of nonlinear time series have been intensively studied. On the basis of the theory of phase space reconstruction, many models have been developed, such as models for local prediction, global prediction, neural network prediction, and fusion prediction that based on basis of radial basis functions [1]. These models have been used for the prediction of sea clutter [2-4] and stock market [5] used to be described by stochastic motion models. Actually these models offer a completely new approach for analyzing complicated phenomena. Similarly, the detected objective signals, in many cases, are always covered by incomplete stochastic background noise during the course of information acquisition and processing. Although the internal noise within the informa- tion processing system always mixes with the detected signal, the frequency bandwidth of all actual system is limited, that is, the signal and noise inside frequency bandwidth are amplified together, but those out of frequency bandwidth are attenuated. Therefore the stochastic noise inside frequency bandwidth is a colored noise whose frequency bandwidth is limited. Studying the colored noise that generally exists in signal processing system has theoretical and practical significance on improving traditional signal acquisition pattern and increasing the ability of detecting weak signal in signal processing system. In this paper, three neural networks are introduced as the prediction models of in-system colored noise time series to verify the predictability of colored noise. The relations between predictability and the frequency bandwidth or sampling rate are discussed as well.

I I . T h e C o l o r e d N o i s e T i m e S e r i e s a n d I t s p r e d i c t i o n M o d e l s

Suppose the computer-generated pseudorandom series {YN } with length N is the time

1Manuscript received date: June 14, 2002; revised date: January 2, 2003.

250 JOURNAL OF ELECTRONICS Vol.20

series sampled at the same time interval. The series naturally contains two time constants, which are sampling interval r and total sampling t ime/Yr. The reciprocals of the two time

constants determine two characteristic frequencies respectively

1 fymax = ~T (1)

and 1

A f = N7 (2)

where fymax is the highest frequency held in the series, A f is the frequency resolution. Assume an ideal band pass filter has frequency response:

0, O < _ ~ d < ~ L

Hd(e j~) = 1, WL < w < wrt (3)

0, h) > W H

where w = 27r f , wrt and o3 L are approximately equal to the upper and lower limit cut-off angular frequencies, respectively.

Through the processing of the band pass filter as Eq.(3), {YN } is translated to {xN}. The highest frequency fz max and the lowest frequency fz rain held in the series {x N } are respectively determined by the upper and lower limit cut-off angular frequency in Eq.(3),

0)H ~)L f , max = ~ - , f , rain = 2-~n" When f , max< fymax or f , min > fymin, {XN} is aco lored

noise time series. In this paper, series {x N } is taken as the sample being studied. For simplification, Bz is defined as the frequency bandwidth held in the series {xN} , Bz ---

fxmax -- fxmin, k is the sampling rate which is relative to f , max, k = 2fymax/fzmax. Based on the theory of phase space reconstruction [6], a trajectory x N = [xN,x N -

T d , X N - - 2 T d , ' ' ' , X N - - ( m - - 1)Yd] w can be constructed in m-dimensional Euclid space by one-dimensional time series {x N } and a suitable delay ra. A mapping can be constructed by the reconstruction of {x N }, and this mapping can be used as a prediction model of the one-dimensional time series. Here we select our prediction models as linear neural network, radial basis function neural network and fusion neural network of radial basis functions and linear functions.

The neural cell function of the linear neural network is a linear function, which means that a simple proportion relation exists between the input and output. In this paper, a linear neural network with single linear neural cells of m inputs are used to construct the prediction model, and m is the dimensions of reconstructed phase space. The output function f ( x ) of the network is obtained from the following equation:

](x) = w x + b (4)

where w = [wx,w2, . . . ,Wm] represents the weight vector; b is the threshold coefficient. In the learning process of the linear neural network prediction model, adjusting the weight

vector and the threshold coefficient can minimize the difference between the output value and the expected value.

The radial basis function neural network consists of an input layer, a hidden layer and an output layer. The output function f ( x ) of its prediction model is expressed as follows:

Nc

] ( z ) = )-~ w~¢j(llx -cjll) (5) j = l

No.4 COLORED NOISE PREDICTION BASED ON NEURAL NETWORK 251

where ¢(r) : R + --+ R is the radial basis functions[Tl, we adopt the popular Gauss function as the radial basis function, that is, ¢(r) = exp[-r2/0.2p], I1" II is the Euclid distance, w j ( j = 1, 2 , . . . , Nc) are weight coefficients, cj (j = 1, 2 , . . . , No), the center of radial basis function, are actually the vectors of sample space which have the same dimensions, Nc is the number of the junction points in the hidden layer, o.p, a selective parameter, determines the width of the center points which are encircled by the radial basis function. No need to adjust the weight values, the input layer of the radial basis function neural network can directly transfer the input signal to the junction points of the hidden layer. The junction points of the hidden layer consisted of the radial basis function ¢(r) : R + -+ R, each junction point of the hidden layer has its own center cj. The hidden layer can transfer to the output layer by adjusting the weight coefficients, and the transfer function of the output layer is a simple linear function. The learning process of the radial basis function neural network prediction model can be divided into two independent steps: the first step is to select a suitable center for each hidden layer; the second step is to minimize the difference between the output value and the expected value by adjusting wj and 0.p.

The prediction model of the fusion neural network of radial basis functions and linear function is constructed by several radial basis function neural networks mentioned above and one linear neural network. This model takes the prediction outputs of several radial basis function neural networks as the input of one linear neural network, fusing the prediction error of every radial basis function neural networks by the linear neural network, and finally a prediction result of a time series can be obtained.

III. p r e d i c t i o n of Co lored N o i s e T i m e Series

The reconstructed vector set {x N } is divided into two groups, {x N lvL }n=l and {XN}nNTNL_b 1. The first NL points are used for learning process of the neural network. According to one- step prediction algorithm, the N T - NL points are used for examining the prediction result of the model. Given the time series {x N NT }n=l, its corresponding prediction series predicted by one of the prediction models is {i N }N__TNL. By applying standard mean-squared devia- tion to describe the prediction precision, the standard mean-squared deviation of one-step prediction is:

1 lvr

N T - N L (xN- N) 2 a2 = n=NL (6}

1 NT

E(x ll = l

where 5 is the mean of {x N NT 0.2 }n=l. When = 0, it means that the prediction results are completely identical with the expect values. If 0.2 = 1, it means that the prediction results are no better than that taking the mean ~ of the series directly Is].

Following the method mentioned above and setting the highest frequency of the series to fmax = fymax/8 , k=8, fxmin = 0, we can obtain {xN} and use it for reconstructing phase space. Suppose delay rd =1 and reconstruction dimension m=5. Take the 400 points of {x N } as experimental data, among which the first 200 points are used for building the model of the neural network, while the last 200 points as the expect values which will be used for examining the prediction result of the model by one-step prediction algorithm. The time wave of prediction result is shown in Fig.1. Fig.l(c) shows the prediction error, which

252 JOURNAL OF ELECTRONICS Vol.20

is the result of Fig.l(a) subtracting the values of Fig.l(b) directly. The scale of vertical axis

is magnified by 100 times. The result shows that the prediction wave is pretty much the same to the expect wave, which illustrates that the prediction error is obviously lower than

the amplitude of original series. We can calculate the standard mean squared deviation and it is a 2 = 1.0855 x 10 -4.

- i

E <

>

m

E <

w e ~

E <

| I I I I I I I I |

0.50 ~

-0.5 -1

-1.5 0 20 40 60 80 100 120 140 160 180 200

Sample points

(a) The time wave of series {x~} ] i I i l i l i i i

0.50

-0.5 -1

-1.5 0 20 40 60 80 100 120 140 160 180 200

Sample points (b) The time wave of the prediction result

0.015 o

0.005 0

-0.005 -0.01 i

0 20 40 60 80 100 120 140 160 180 200 Sample points

(el The time wave of the prediction error

Fig.1 The prediction result of the colored noise by the linear neural network

For further studying the predictability of the colored noise, suppose the highest fre-

quency of the series is fy max, fy max/2, fy max/4, fy max/8, fy max/16, fy rain • 0, respectively, then the relative sampling rate k is 2, 4, 8, 16, 32, respectively. Let delay Td=l, reconstruc- tion dimension m = 5 and m = 10. Still take the 400 points of {x N } as experimental data, the first 200 points are used for model building, the last 200 points for expect values. Input the reconstructed vector set into the linear neural network, the radial basis function neural network and the fusion neural network, respectively. When the relative sampling rate takes

different values, the prediction error values are listed in Tab.1. It shows a rule that the pre- diction error for the colored noise decreases with the increment of the relative sampling rate. Increasing sampling rate for the colored noise means that the density of observed points in every signal cycle is increased, so it will decrease randomness, and the prediction results get better. When the relative sampling rate k is 2, actually it is the original stochastic series {YN } that is predicted; therefore the error is greater than 1, which conforms to the fact that complete stochastic series has no predictability in general.

No.4 COLORED NOISE PREDICTION BASED ON NEURAL NETWORK 253

T a b . 1 T h e p r e d i c t i o n error w h e n t h e r e l a t i v e s a m p l i n g ra te is d i f ferent

m = 5 m = 1 0

k L i n e a r ~r 2 R a d i a l b a s i s ~2 F u s i o n a 2 L i n e a r a 2 R a d i a l b a s i s a 2 F u s i o n cr 2

2 >I >I >i >I >i >I

4 0.0472 O. 1409 0.0506 0.0090 0.4070 0.0750

8 1 . 0 8 5 5 x 10 - 4 1 . 3 7 1 6 x 1 0 - 4 1 . 4 0 3 7 x 10 - 4 6 . 2 5 8 9 × 1 0 - 6 2 . 6 3 7 7 × 1 0 - 5 1 .1868)<10 - 5

16 1 . 7 4 0 9 x 10 - 7 2 . 1 9 8 3 × 1 0 - 7 1 . 8 0 7 3 x 10 - 7 1 . 2 9 6 1 x 10 - s 1 . 2726X 10 - 7 2 . 5 2 6 6 × 1 0 - 8

32 2 . 0 4 6 5 × 1 0 - 9 8 . 0 3 3 4 X 1 0 - s 5 . 5 1 4 3 x 1 0 - 9 2 . 1 8 4 4 x 1 0 - 1 ° 1 . 6 0 7 3 x 1 0 - 8 2 . 7 6 4 7 x 1 0 - 9

In addition, the prediction results of the series {x N } is investigated when the filter band- widths Bx takes different values. Set the highest frequency of the series fxmax = /ymax, the relative sampling rate k=2. By changing fxmin, thus setting the relative frequency bandwidth Bx//~max to 1, 1/2, 1/4, 1/8 and 1/16, respectively, keeping delay Td=l, recon- struction dimension m = 5 and m = 10, the data and experimental approaches are the same as those mentioned above. Tab.2 shows the prediction error values relate to the different relative frequency bandwidths. This time we conclude that the prediction error decreases with the narrowing of the relative frequency bandwidth.

T a b . 2 T h e p r e d i c t i o n error w h e n t h e r e l a t i v e f r e q u e n c y b a n d w i d t h is d i f ferent

m = 5 m = 1 0 B ~

.f~ m a x 1

L i n e a r a 2 R a d i a l b a s i s a 2 F u s i o n a 2 L i n e a r a 2 R a d i a l b a s i s 62 F u s i o n 0 -2

>i >i >I >I >I >i

1 / 2 0 . 0 1 0 4 0 . 0 0 8 0 0 . 0 1 0 0 0 . 0 0 5 5 0 . 0 0 8 9 0 . 0 0 5 6

1 / 4 0 . 0 0 4 8 0 . 0 0 5 7 0 . 0 0 6 4 0 . 0 0 2 6 0 . 0 0 3 9 0 . 0 0 3 2

1 / 8 3 .6448>(10 - 4 5 . 1 9 8 7 x 10 - 4 3 . 0 4 9 3 x 10 - 4 2 . 6 3 3 4 x 10 - 4 4 . 2 7 0 9 x 10 - 4 2 . 5 8 3 3 × 1 0 - 4

1 / 1 6 5 . 1 1 0 0 x 10 - 5 8 . 2 3 6 9 x 10 - 5 4 . 8 4 6 8 x 10 - 5 4 . 3 3 8 3 x 10 - 5 6 . 0 0 6 3 x 10 - 5 3 . 8 7 2 6 x 10 - 5

The results of the Tab.1 and Tab.2 indicate that the three neural networks all can predict the colored noise, and the prediction results have little to do with the transfer function or the structure of the neural networks. The results of many experiments also show that the value of the reconstruction dimension m has some influence over the prediction results, i.e., the prediction error will increase if the value of m is either too large or too small. With proper values for m, k and Bx, the prediction error can meet the needs of the real engineering in quantity.

IV. Conclusions

So far, existing research results indicate that complete stochastic noise is unpredictable in general. This paper reveals that the colored noise is indeed predictable. The prediction error decreases with the increase of the relative sampling rate for the colored noise and with the narrowing of the filter bandwidth for stochastic noise, but almost has nothing to do with the transfer function or the structure of the neural networks.

Further study of the cause of the predictability presented in the colored noise will be helpful for revealing the intrinsic quality of colored noise.

Based on the average power or variance of noise, traditional signal processing theory built the theories and approaches for extracting signal from noise only when the signal's

254 J O U R N A L OF E L E C T R O N I C S Vol.20

power is comparable with tha t of noise. The research in this paper is based on the real-time

na ture of s tochast ic noise. The real-t ime predict ing and a t tenuat ing of the noise in this work

is undoubted ly a way to raise the SNR of the sys tem in essence. By increasing the SNR,

the applicabili ty of the est imation theory will be extended to the sys tem within which the

sys tem noise is much larger than the detected signal, thus the detect ion of some information,

which is hard to be detected before, can be achieved.

R e f e r e n c e s

[1] Cheng Qianshang, Wu Lianwen, et al., Fusion prediction by a attribute cluster network and radial

basis function, Chinese Science Bulletin, 45(2000)11, 1211-1216, (in Chinese).

[2] H. Leung, Experimental modeling of electromagnetic wave scattering from an ocean surface based on

chaotic theory, Chaos, Solitons and Fractals, 2(1992)1, 25-43.

[3] H. Leung, Chaotic radar signal processing over the sea, IEEE Journal of Oceeanic Engineering, 18(1993)3,

287-295.

[4] R. P. Chakravarthi, Radar target detection in chaotic clutter, Proceedings of the 1997 IEEE National

Radar Conference, Syracuse, New York, IEEE Aerospace and Electronic Systems Society, 1997, 367-

371.

[5] Zeng Zhaocai, Duan Yurong, et al., Chaotic time series analysis based on radial basis function net-

works, Journal of Chongqing University(Natural Science Edition), 22(1999)6, 113-120, (in Chinese).

[6] F. Takens, Detecting strange attractors in turbulence, Lecture Notes in Mathematics, Berlin, Spring,

1981, 898, 366-381.

[7] J. Moody, C. Darken, Fast learning in networks of locally tuned processing units, Neur. Comp.,

1(1989)2, 281-294.

[8] M. Casdagli, Nonlinear prediction of chaotic time series, Physica D, 1989, 35,335-356.