Proceedings of Intemational Joint Conference on Neural Networks, Montreal, Canada, July a1 - August 4, 2005

Issues in Designing AutomatedMinimal Resource Allocation Neural Networks

Momcilo MarkusIllinois State Water SurveyChampaign, Illinois 61820

E-mail: momcilo@sws.uiuc.edu

Abstract - Artificial Neural Networks (ANNs) have a long recordof generally promising results in hydrology. The earlierapplications were mainly based on the back-propagation feed-forward method, which often used a lengthy trial-and-errormethod to determine the final network parameters. An attemptto overcome this shortcoming of the traditional applications is theMinimal Resource Allocation Network (MRAN). MRAN is on-line adaptive method which automatically configures the numberof hidden nodes based on the input-output patterns presented tothe networlk Although MRAN demonstrated superior accuracyand more compact network, when compared with the traditionalback-propagation method, some additional questions need to beaddressed. While the number of hidden nodes is estimatedautomatically, other user-defined parameters are selectedarbitrarily, and adjusted through simulations. This researchaddresses determining the user-defined parameters prior to themodel run. The research also compares MRAN results from twoapplications, and discusses a pathway towards designing a fullyautomated MRAN.

I. INTRODUCTION

Applications of Artificial Neural Networks (ANNs)in hydrology range from rainfall-runoff modeling, streamflowprediction, precipitation forecasting and groundwatermodeling to water resources management and water qualitymonitoring. It appears that the Back-Propagation (BP) [8] isthe most frequently used hydrologic application, particularlyin earlier applications ofANNs ([2], [3], [11]). BP is based ona gradient descent method and uses the first-order partialderivatives of the error function in searching for the optimalparameter set for the network. This learning techniquerequires a priori knowledge of the number of hidden layersand the number of neurons in each layer, meaning that thecomplete architecture of the network must be defined inadvance.

Another class of artificial neural networks used inhydrologic applications is the radial basis function (RBF)neural networks [5]. The RBF networks [4] use radiallysymmetrical basis functions (usually Gaussian) as theiractivation functions are different than the sigmoidal functions.As in the back-propagation networks, most algorithms

traditionally used for training the RBF neural networks requirethe number of hidden units to be fixed a priori by a trial-and-error procedure.

The (MRAN) algorithm [12] is a techniquedeveloped for training of RBF neural networks. It assigns thenumber of hidden units automatically based on growth criteria,and also prunes inactive hidden neurons. It uses ExtendedKalman Filter (EKF) to recalculate the model parameters ineach step. MRAN (Figure 1) is a sequential learningalgorithm for Gaussian RBF neural networks [12].

MRANs have network parameters and user-definedparameters. Network parameters are calculated by thealgorithm, and they include the number of hidden units (h), thecenters (j) and the widths (a) of the hidden units, the weights(a), and the biases (ao). They are dynamic as their totalnumber change at each iteration. Values of the networkparameters generally determine the magnitudes of the modeloutput. On the other hand, the user-defined parameters needto be defined a priori to run the model and include thethresholds for adding hidden neurons (Es, £2, and £), thresholdfor pruning hidden neurons (6), various coefficients (7y, K, po,and qo), and the size of the sliding data window (sj). Theydetermine the accuracy of the model output. User-definedparameters remain constant throughout the iteration period.

II. MODEL APPLICATION

Performance of the MRAN algorithm is evaluatedusing hydrometeorological and water quality data for theUpper Sangamon River in Central Illinois, with a drainagearea of about 2,400 square kilometers. Accurate prediction ofrainfall-runoff and nitrate-N concentration is of a greatimportance for agricultural watersheds in north-central Illinois([7], [9], [13]). Performance of the MRAN algorithm wasevaluated at the outlet of the Upper Sangamon Riverwatershed based on daily and weekly nitrate-N forecasting. Inboth cases MRAN exhibited more accurate forecasts than BP.Details on daily forecasts can be found in [1], and weeklyforecasts are presented in [6] and [7].

0-7803-9048-2/05/$20.00 @2005 IEEE 2671

p

TABLE I

USER-DEFINED MRAN PARAMETERS

Symbol Description of the variable

q Number ofnetwork inputsp Number ofnetwork outputsC Inm Max threshold for criterion #I for adding neurons

S-imim Min threshold for criterion #I for adding neurons

Sy Decay constant for criterion #16 Threshold for pruning neuron92 Threshold for criterion #2 for adding neuron£3 Threshold for criterion #3 for adding neuronv Overlap factor

qo Coefficient for random walk in EKF

Po Estimate ofparameter uncertainties in EKFnw Size of sliding window for adding neuronsSW, Size of sliding window for removing neurons

III. DIscussioN

Figure 1. Flowchart ofMRAN algorithm

In the MRAN algorithm there are no establishedguidelines for assigning values to the user-defined parameters.Generally, these parameters are specified through trial-and-error and it is usually difficult to estimate the initial values.Preliminary ranges for the user-defined parameters wereestablished based on the above described daily and weeklyapplications (Tables 1 and 2).

Output accuracy and computational time both dependon the values of user-defined parameters and whetherstandardized data or raw data are used. If data transformationor data standardization change the data range, selection ofthresholds should be modified accordingly. For example,selection of a large sliding window often could decrease thenumber of hidden neurons, and hence computational time, butit also could reduce output accuracy.

The MRAN algorithm was found to be very sensitiveto the thresholds (&1, 62, s3, and 8), and the parameter qo in theEKF. Based on sensitivity analysis and values adopted inprevious studies [10], the recommend parameter ranges foreach parameter are presented in Tables 1 and 2. Table 1provides definition for each parameter, while Table 2 showsthe actual ranges for parameters.

Figure 2 shows predictions of weekly nitrate-N data onthe Upper Sangamon River near Monticello, Illinois, usingMRAN. The forecasting accuracy exceeded that of the BP([7], [1]). The variations in the number of nodes in Figure 2show increased complexity, and thus higher number of nodes,during summers, and decreased complexity and lower numberof nodes during winters. MRANs were sensitive to changes inthresholds for adding or pruning neurons. Also, MRANsdemonstrated high sensitivity to size of sliding windows foradding or removing neurons (Table 2).

Traditional statistical methods often have input selectioncriteria embedded in the algorithm. Those criteria generallycompare benefits of including additional variable/lag with theincrease in model uncertainty. On the other hand neither BPnor MRAN in their original form have a module for automaticinput selection. Those algorithms would behave similarly forvery weakly and very strongly correlated input-outputsystems. This could be overcome by designing a combinedmethod as presented in Figure 3, where the Statistical Methodsblock automatically selects relevant inputs, and the EmpiricalMethods selects the user-defined parameters. Examples ofcommonly used statistical criteria are AIC, BIC, and cross-entropy.

2672

TABLE 2

VALUES FOR THE USER-DEFINED PARAMETERS

Weekly Weekly DiyMSSymbol sWsl IO DailyMISO

q 1 4 4P I I ICIMXl 1.20 2.20 2.0-4.5CImm 1.0 0.30 0.5-3.0y 0.5 0.999 0.8-1.0

0.4 0.05 0.001-0.5C2 0.39 0.10 0.01-0.5£3 0.425 0.30 0.02-.5K 0.84 0.88 0.5-1.0qo 0.003 0.005 0.001-.5Po 0.090 0.90 0.85-1.0n, 8 15 20-100SW 3 1 5 20-100

Comparison of model output and observed data - for taning period4

- Obsrved I I

2i, - inimt

0 50 100 150 200 250 30010,a

c

Etz X

IIIII ~ ~ ~ ~ ~~II

~ ~ ~~I

0SO 100 150 200 250 3009,

0

I-

0 50 100 150 200 250Time (weeks)

Figure 2. Predictions of weekly nitrate-N data on the Upper Sangamon Rivernear Monticello, Illinois, using MRAN

IV. CONCLUSIONS

This study introduced the MRAN algorithnm, whichdemonstrated the potential for overcoming shortcomings ofthe traditional back-propagation algorithm. The algorithmwas evaluated for hydrologic applications through two casestudies. The algorithm was applied to the Sangamon Riverwatershed in central Illinois to predict of streamflow andnitrate-N concentration, and to simulate nitrate-N data. TheMRAN algorithm was compared with the back-propagationalgorithm and demonstrated improved accuracy. Directions ofthe future improvements ofthe method were discussed.

Figure 3 Network with input selection module and a priori estimation ofinitial parameters

V. REFERENCES

[I] Amenu, G., M. Markus, P. Kumar, and M. Demissie, (2005), Hydrologicapplications of Minimal Resource Allocation Network (MRAN), J. Hydrol.Eng. (under revision)[2] ASCE Task Committee, Artificial neural networks in hydrology - I:Preliminary concepts, J. Hydrologic Eng. 5(2), 115-123, 2000a.[3] ASCE Task Committee, Artificial neural networks in hydrology - II:Hydrologic applications, J. Hydrologic Eng., 5(2), 124-137, 2000b.[4] Broomlhead, D.S, Lowe, D., (1988) Multivariable Functional Interpolationand Adaptive Networks, Complex Systems, vol. 2, pp. 321-355.[5] Govindaraju, R.S., and B. Zhang, Radial basis function networks, inArtificial Neural Networks in Hydrology, edited by R.S. Govindaraju and A.R.Rao, pp. 93-109, Kluwer Academic Publishers, The Netherlands, 2000.[6] Markus, M. ANN in Hydrology, in Kumar, P., Alameda, J., Bajcsy, P.,Folk M., and Markus, M., Hydroinformatics - Data Integrative Approaches,CRC Press, Boca Raton, publ: Oct. 2005.[7] Markus, M., C. W.-S. Tsai, and M. Demissie, Uncertainty of weeklynitrate-nitrogen forecasts using artificial neural networks. J. EnvironmentalEng., 129(3), 267-274, 2003.[8] Rumelhart, D.E., and R.J. Williams, Learning internal representations byerror propagation, Nature, 323, 533-536, 1986.[9] Suen, J.-P., and J.W. Eheart, Evaluation of neural networks for modelingnitrate concentrations in rivers, J. Water Resour. Plan. Mang., 129(6), 505-510,2003.[10] Sundararajan, N., Saratchandran, P., and Ying Wei, L., Radial BasisFunction Neural Networks with Sequential Learning, MRAN and ItsApplications, World Scientific, Singapore, 1999.[11] Tokar, A.S. and M. Markus, Precipitation-runoffmodeling using artificialneural networks and conceptual models, J. Hydrol. Eng., 5, 156-161, 2000.[ 12] Yingwei, L., N. Sundararajan, and P. Saratchandran, Performanceevaluation of a sequential minimal radial basis function (RBF) neural networkalgorithm, IEEE Trans. Neural Networks, 9(2), 308-318, 1998.

2673