NAFIPS 2005 - 2005 Annual Meeting of the North American Fuzzy Information Processing Society

Enhanced Multivariable TS Fuzzy Modeling in Neural Network Perspective

Ozer Ciftcioglu and I.S. SariyildizDelft University of Technology, Faculty of ArchitectureBuilding Technology, 2628 CR Delft, The Netherlands

o.ciftcioglu(Abk.tudelft.nl; Tel:31-15 2784485; Fax:31-15 2784127

Abstract -A new approach is presented to enhance fuzzymodeling using multidimensional fuzzy sets directly in the fuzzymodel in place of decomposed fuzzy sets by projection. Theantecedent fuzzy sets in the form of multivariable functions arerepresented in continuous form. This is accomplished by using amulti input and multi output neural network, which is inparticular radial basis functions (RBF) network providing therequired multivariable function approximation properties in afuzzy model. The inputs of the network are fuzzy model inputs,outputs are the membership function values as to themultivariable fuzzy sets involved in the model. Thus, each outputis restricted to one multivariable fuzzy set. The RBF network istrained according to the multidimensional fuzzy sets which areidentified after fuzzy clustering of the data. Based on this, thelinear model parameters are determined by the method of leastsquares in a straightforward manner. The transparency issuescan be handled by means of projection process to haveinformation about the shape and locations of the membershipfunctions pertinent to each variable at the input. The accuratefuzzy modeling having been thus guaranteed, the information onthe linear model parameters are used to establish themultivariable fuzzy rules with their respective validity regions.The redundancy of the number of fuzzy sets can be circumventedby classical cluster merging algorithms available. In thisapproach, the fuzzy rules are determined from the local linearmodels, however in contrast with the conventional fuzzymodeling, the model outputs are determined by the multivariablefuzzy sets. To obtain regular sets, RBF network is trained bymeans of orthogonal least squares (OLS) method so that at thesame time the convergence issues of general neural networktraining is eliminated.

I. INTRODUCTION

Fuzzy modeling techniques have been applied to nonlinear,uncertain and complex systems. A number of researches havebeen reported in the literature in a variety of disciplines ofexact sciences and soft sciences. One of the aspects that isclaimed for fuzzy modeling is that it distinguishes itself fromother black-box approaches like neural networks as fuzzymodels are to a certain degree transparent to interpretation andanalysis. However absence of a clear procedure for generalfuzzy modeling already is a firm indication of the complexityof the task providing transparency. Nevertheless, for certaintype of fuzzy modeling rather well-established methods areavailable for practical applications. One of such modelingtechniques is known as the method of Takagi-Sugeno (TS)model in fuzzy systems literature. It has been one of the majortopics in theoretical studies and practical applications of fuzzymodeling and control. The basic idea of this method is todecompose the input space into fuzzy regions and toapproximate the system in every region by a simple model.

The TS model is usually identified in two steps. First, thefuzzy sets in the rule antecedents are determined, producingpartitioning of the input space. Then, when the ruleantecedents are fixed, the consequent parameters aredetermined. Since, in most cases the knowledge about thesystem being modeling is not enough to place the fuzzy setsand consequent parameters, data driven models areubiquitously used. For this purpose least square estimationfrom the observation data is commonly used. This can beemployed in two ways. One solves a local, weighted least-squares problems, one for each rule. The other solves a globalleast-squares problem. Local least squares gives more reliablelocal models, while global least squares gives a minimalprediction error estimate. In determining the antecedents fuzzysets, in the multivariable input case, one clusters the data toobtain multivariable fuzzy sets and by means of projection oneach input variable one obtains the respective fuzzy sets.However, during the projection process an irrecoverableinformation loss occurs so that, the fuzzy modeling from theprojected fuzzy sets becomes a point of concern. As to onedimensional input case, there is no need for projection andtherefore the model is satisfactorily accurate. This explains thereason why the majority of fuzzy modeling applicationsconcern merely this simple case. However, in two-dimensional case, the model becomes less precise and thequality of the model exponentially aggravated with theincrease of the number of fuzzy sets employed in eachdimension. Note that this is not only because of curse ofdimensionality but also because of information loss due todecomposition of the multivariable fuzzy sets. The situationbecomes already a serious issue even in low dimensional inputcase which can be as low as two. This explains the fact that,the majority of the fuzzy logic applications use at most two-dimensional input space and not more although the vastnumber of theoretical studies reported may extends to anydimension without actual reference to the information losswhich occurs in the projection process. As to curse ofdimensionality of the fuzzy sets, higher dimensional fuzzymodeling become extremely cumbersome and contradicts theclaimed transparency and practicality of fuzzy logic.

After the general observations above having been made,another issue in data-driven modeling is the size of the datasamples. Namely, data-driven fuzzy modeling is moreinteresting compared to neural networks due to itsinterpretability of the fuzzy membership functions. Otherwise,if there is enough data able to train a neural network, thenneural network is a competitive alternative to fuzzy modelingin case the transparency is a marginal issue. In such cases,

O-7803-9187-X/05/$20.00 ©2005 IEEE. 150

universal approximation property of fuzzy modeling isconsidered apart from the interpretability properties of themodel. However, in fuzzy modeling, the multivariablemembership functions provide rich aggregated information ofdata samples. Using this information directly one obtainsrelatively more accurate results with relatively less number ofdata samples without any sacrifice in transparency. At thisvery point, fuzzy modeling gains due attention relative toneural network models. In this research, for the desiredenhancement of fuzzy model, the multidimensionalmembership functions are modeled by means of a neuralnetwork of a radial basis functions (RBF) type. Hence, theoutcome of a multivariable fuzzy model is based on thiscompact fuzzy model. For the transparency, one of theconventional decomposition schemes can be used. That is, forthe sake of minimized model prediction error by the presentapproach, transparency is not diminished as trade-off, as thisis explained later in the paper. The motivation of this researchis due to fuzzy modeling of architectural design data for thedevelopment of intelligent design processes where the data aremultidimensional and highly nonlinear. Even in a simple case,if the right parameters are not given, the fuizzy modelingdegrades rapidly, thereby the transparency. Therefore theconventional fuzzy modeling approach in a multidimensionalcase may easily be complex and consequently inaccurate sothat the model enhancement becomes most desirable for suchadvanced modeling tasks. This article aims for this with themotivation of applying fuzzy logic to soft sciences, inparticular Architecture, in this enhanced form. Although bothfuzzy logic and soft sciences are well established, the directinteraction of these technologies is not commonly exercised.The reason for this might partly be due to rich mathematicalbackground behind fuzzy logic making direct access difficultfor users, which are not in the same field. Conversely,someone from engineering sciences does not easily realize theproblems subject to interest in soft sciences. Because of thisinconvenience, the pace of fuzzy logic employment in softsciences is rather slow in a general sense. With the advancesin interdisciplinary research, it is almost certain that, fuzzylogic is going to play an important role in informationprocessing tasks of soft sciences by the coordination ofvarious efforts from different fields. Today, these types ofactivities are rapidly increasing thanks to advances in theinformation and communication technology known as ICT.The organization of the paper is as follows. Section two givesbrief description of TS fuzzy modeling and make reference toRBF network and OLS training algorithm applied to RBFnetwork for modeling the multivariable fuzzy membershipfunctions by this network. Section three presents thecomparative results obtained using different multivariable dataincluding one from the literature. This is followed by theconclusions in section four.

II. FUZZY MODELING

A. Takagi-Sugeno Fuzzy Modeling

Takagi-Sugeno type fuzzy modeling [1] consists of set offuzzy rules a local input-output relation in a linear form asR5 :If xi is 4A and...xn is An (1)

Then yA =ax+b,, i= L,2 ......,Kwhere R, is the ith rule, x=[xl,x2,x]Te_ .is the vector ofinput variables; Ail, Ai2,... Ain are fuzzy sets and y, is the ruleoutput; K is the number of rules. The output of the model iscalculated through the weighted average of the ruleconsequents of the form

K

,(xAYi (2)A i=1

iK

In (2), P,(x) is the degree of activation of the ith rulen

g,(x)JJ/IuA (xj), i=1,2,....,K (3)

where PA (xj) is the membership function of the fuzzy set

Ai, at the input (antecedent) of Ri. To form the fuzzy systemmodel from the data set with N data samples, given byX=[XI X2, .. . XN]T, Y=[Y1 Y2 . YN] (4)where each data sample has a dimension of n (N>>n). Firstthe structure is determined and afterwards the parameters ofthe structure are identified. The number of rules characterizesthe structure of a fuzzy system. The number of rules isdetermined by clustering methods. Fuzzy clustering in theCartesian product-space -ExW is applied to partition thetraining data. The partitions correspond to the characteristicregions where the system's behaviour is approximated by locallinear models in the multidimensional space. Given thetraining data F and the number of clusters K, a suitableclustering algorithm [2] is applied. One of such clusteringalgorithms is known as Gustafson-Kessel (GK) [3]. As resultof the clustering process a fuzzy partition matrix U isobtained. The fuzzy sets in the antecedent of the rules areidentified by means of the partition matrix U which hasdimensions [NxK]; N is the size of the data set. The ik-thelement of gtikE [0,1] is the membership degree of the i-th dataitem in cluster k; that is, the ith row of U contains the pointwise description of a multidimensional fuzzy set. One-dimensional fuzzy sets Ai. are obtained from themultidimensional fuzzy sets by projections onto the space ofthe input variables xj. This is expressed by the point-wiseprojection operator of the form/1A4 (Xjk) = pro;j (Pik) (5)The point-wise defined fuzzy sets Ai. are then approximatedby appropriate parametric functions. The consequentparameters for each rule are obtained by means of linear leastsquare estimation. For this, consider the matrices

X=[Xl,..., xN]T, Xe[Xl] (extended matrix [Nx(n+1)] ); A(diagonal matrix dimension of [NxN]) and

XE=[(AiXe); (A2Xe);...... (AKXe)] ([NxK(n+1)]) (6)

where the diagonal matrix Ai consists of normalizedmembership degree as its k-th diagonal element

151

Ax (xk) (7)flAi(Xk) =

E fj (X,)i=1

The parameter vector to dimension of [Kx(n+1)] is given by=[*1 ....... OK%] (8)

where OiT=[a,T b1] (1< i < K).Now, if we denote the input and output data sets as XE and Yrespectively then, the fuzzy system can be represented as aregression model of the matrix formY=XE i+e (9)

D. Multivariable Fuzzy Membership Functions Modeled byRBFNetwork

As result of clustering process, the multivariable fuzzymembership functions (MF) are obtained by point wiseprojections in the multidimensional space, as described above.The clustering method can be supplemented with a clustermerging algorithm, so that appropriate number of clusters withassociated multidimensional membership functions areobtained. Note that, the redundancy in the number ofmembership functions may cause serious detrimental effectson the model being developed. As result of clustering process,next to the fuzzy partition matrix, also the cluster meansmatrix, cluster covariance matrices, the eigenvalues, and theeigenvectors of the covariance matrices are obtained. Fromthese also the TS model parameters are obtained. Namely,considering the case of two time-discrete input signals x,(k)and x2(k) and one time-discrete output signal y(k) representingthe system output, TS fuzzy models are characterized by rulestatements of the general formIF x, (k) is A(i) AND X2 (k) is BTHEN

y(k) = a(r) x, (k) + b") x2 (k) + c')where i,j,r=l,...,n,Above, A(i) and B(i) represents fuzzy sets of the input variablesxi and x2, and a( ) b(r ) and c(r) ) are the coefficients of theoutput models. n, is the number of clusters

As a novel approach in this research, first, TS fuzzy modelby projections is established for transparency by means ofglobal learning. Second RBF network is established thatmodels the multivariable membership functions. RBF networkhas been extensively studied due to its simple architecture andfast learning [4,5]. RBF network belongs to group of kernelfunction network that utilizes simple kernel functions. Thearchitecture consists of one hidden and one output layer. Eachhidden node represents one of the kernel functions. An outputnode generally computes the weighted sum ofthe hidden nodeoutputs. A kernel function is a local function and the range ofits effect is determined by its center and width. The Gaussianfunction is a popular kernel function and used in this work.The training of the RBF network is done by OLS [5,6]

algorithm. For an RBF network to perform trainingsuccessfully, the number of clusters or the number of hiddenlayer nodes has to be optimum. An optimum value for thisnumber is generally not known. The OLS algorithm treats theselection of representative basis functions as a subset selectionproblem. The fuzzy model inputs as data samples are providedat the RBF network input and the corresponding membershipdegrees of these samples are provided at the RBF networkoutput. For example, for two model variables x=[xl x2] andfive clusters, the RBF network has two inputs and fiveoutputs. Based on the membership degree information theoutput polynomial information, i.e., a() b(r) and c(r) obtainedfrom the result ofGK clustering and global learning. Namely,the local linear model coefficients can be obtained by usingthe cluster centers and the eigenvectors of the covariancematrices information thereby identifying the fuzzy model.Alternatively, the model parameters can be obtained usingmultidimensional fuzzy membership information and themethod of global least squares for learning.

III. COMPUTER EXPERIMENTS

Investigations are carried out by means of simulationwhere the data set subject to fuzzy modeling belongs to aMISO system with two inputs and one output. The systemsubject to modeling is described by the function given byy(t) = F(x, (t),x2 (t))With the a priori unknown nonlinear mapping

F(x1,x2) = sin[R(x,,x2)]1R(x,1x2)(10)where

R(x1,X2) = (X, 10)2 + (X2 _ 10)2

This highly nonlinear function is known as 'sombrero' andshown in figure 1. It is rather suitable surface for testing theapproximation carried out by fuzzy modeling.

rlg.1 ine mgniy noiiimear umncuon somorcro , suojeci to mLzzymodeling

After providing the system with the harmonic input signals

152

x(t) = 10 [1 + cos(2tt)] (11)

y(t) = 20 sin 0-tfor the dimensionless time range 0< t < 10, the N=105 timediscretely measured data records with sampling time interval0.095. The input and the output values ofthe system constitutethe basis of the fuzzy modeling procedure. The result of theGK fuzzy clustering algorithm with 8 clusters is illustrated infigure 2 where the system-specific surface formed by the dataof the input product space and the output space, as well as thepositions ofthe identified cluster centers are shown.

Fig. 2 System specific surface and six clusters and product space ofthe input variables xl and x2 (uppermost as contour plot and the

lowermost plot)

The computer experiments of fuzzy modeling are carriedout in two ways for comparison. Firstly, the model isestablished via projected membership functions inconventional way as described in section II. The fuzzy MFsobtained by means of pointwise projection and with smoothgaussian-shaped functions are shown in Fig.3. The clusters ofthe projected fuzzy membership functions are seen in figure 2.The output of the fuzzy model established from the MFswhich are shown in figure 3, are presented in figure 4, wheretwo plots are given. The line with the connected dots is themodel output. There is a marked difference between the dataplot and its model output counterpart. This difference isattributed to the information loss due to projection process.Three more computer experiments have been carried outsimilar to that shown in figure 4, to test the modelperformance. However, these are distinguished by providingthe system with three different harmonic input signals otherthan that given in (11). These input signals are given byx(t) = 10 [1 + cos(2ra)]

y(t) = 20 sin t) 210 samples (12)

x(t) = 10 [1 + cos 7)]

y(t) = 15 sin -t(10 )x(t) = 10 [1 + cos ntt)]

y(t) = 20 sin -tKb0 )

105 samples (13)

105 samples (14)

The corresponding model outputs together with the actualones are presented in figure 5.

s MLemnersmnp iunctlons ontameu Dy projectionmultivariable clusters seen in figure 2.

Fig. 4 Fuzzy model output (dotted) and the true counterpart plots forthe harmonic input signal in (2)

In figure 5, uppermost plot is for 210 samples half ofwhich is the same data samples used for the model formationsince only the sampling interval is halved as 0.095/2.Therefore the differences between the actual values and themodel outputs (dotted lines) are relatively limited. In themiddle plot, deviations are high, because the data samples are

153

different than those used for the modeling as (13) indicates.The lowermost plot in figure 4 is obtained using the signalsgiven in (14) and they are similar to those given in the middleplot as one expect, since the data samples are again differentthan those used for the modeling.

Fig. 5 Fuzzy model output (dotted) and the true counterpart plots forthe harmonic input signal in (3)-(5)

The results presented in figure 5 clearly indicate that the fuzzymodel is not satisfactory, in this case. This is attributed tohighly nonlinear system and projected membership functionsused for modeling. To test the model performance further forcomparison purpose later with the model obtainedmultivariable fuzzy membership functions, data samples,which are not used before for modeling are applied as inputsto the fuzzy model. These are selected as 20 by 20 grid pointson the xl-x2 plane so that the total number of samples is 400.The test data samples and the and their true counterparts areshown in figure 6.

Fig. 6 Fuzzy model output and the true counterpart plots for theharmonic input signal in grid form of 20x20.

In figure 6 the upper 2D plot and the corresponding 3D plotindicate rather unsatisfactory model performance as thedeviation between true data samples and the model response israther high. The 'sombrero' is not to recognize indicating theunfavorable modeling conditions exhibited by the 'sombrero'shaped system surface. The identified surface in the lower plotis not to recognize as 'sombrero', however as it should.The same experiments the results of which reported in figures4-6 are carried out with the multidimensional fuzzy sets usingthe RBF network model of these sets, as described in sectionII. The results are presented in figures 7-9.

Fig. 7 Fuzzy model output (dotted) and the true counterpart plots forthe harmonic input signal in (2)

Fig. 8 Fuzzy model output (dotted) and the true counterpart plots forthe harmonic input signal in (2)

Figure 7, shows the three different outputs of the fuzzymodel for the corresponding input signals given by (12)-(14),respectively. These results are to compare with those in figure5 and the apparent improvement is to observe. In figure 8, theoutput of the fuzzy model with the multivariable membershipfunctions are better following the actual counterpart relative tothat seen figure 4 where the model is obtained by means of the

154

projected membership functions. The improvement in figure 7relative to figure 5 is attributed to the multivariable fuzzymembership functions directly used for making the fuzzymodel.

Fig.9 Fuzzy model output and the true counterpart plots for theharmonic input signal in grid form of20x20.

In the same way, figure 9 shows the output of the modelwith multivariable membership functions with the actualcounterparts using data samples on a 20 by 20 grid on the xl-x2 plane. Figure 9 is to compare with figure 6. As to figure 9,a marked improvement is observed relative to that given infigure 6. The persistent improvements in figures 7-9 areattributed to the multivariable membership functions directlyused in fuzzy model formation. The basic explanation of thismodeling degradation mentioned above is partly due toirrecoverable systematic information loss during thedecomposition process of the multidimensional fuzzy sets viaprojection. Another source of systematic error is due toreconstruction of the multivariable membership ftmctionsfrom the covariance information of the clusters and norm-inducing matrix during the recall process of the model [7]. Byworking multidimensional fuzzy sets, such systematic errorsare eliminated. However, using the multivariable fuzzy setsdirectly, the transparency of the model diminishes. To recoverthis, still the common projection process can be applied tomaintain it via common fuzzy modeling using projectedmembership functions as to figure 3. It is noteworthy tomention that, in figure 3, seemingly eight membershipfunctions for each input dimension can be aggregated toreduce the complexity of the model. In the present case thereduction of complexity by aggregation is not investigated.Instead, the seemingly redundant fuzzy membership functionsare eliminated making them less than eight in number.However, this act diminishes the model performance, which isalready below the satisfactory level with eight membershipfunctions. This observation is attributed to the high-nonlinearity of the surface and therefore need of moremembership functions for representing the details of thesurface.

IV. CONCLUSIONS

Fuzzy modeling with multidimensional fuzzy sets ispresented. Conventional fuzzy modeling with decomposedfuzzy sets gives satisfactory performance for slightlynonlinear specific system surface as this is demonstrated in theliterature [8,9]. However for highly nonlinear surfaces in amultidimensional space, the same modeling process degradesand the approach needs to be supported for enhancement.Such desired enhancement is accomplished in this work bymeans of integrating an RBF network to the modeling process.The RBF network plays the role of representingmultidimensional fuzzy membership functions. In particular,RBF networks have their origin in the solution of themultivariate interpolation problem. It has been shown that, incompact domains the function space realized by RBF networkis dense in the space of all continuous functions [10], that is, itis universal approximator of continuous fumctions. Therefore,after establishing the fuzzy model, in the recall phase, theform of the membership functions remain the same; that isthey are not recalculated from the covariance information, asthe recalculation is performed in common fuzzy modeling andit introduces some systematic error on the estimation at theoutput of the model. The fuzzy model having beenestablished, the RBF support especially counts during theactual utilization of the fuzzy model that is in the recall phase.

V. REFERENCES

[1] T. Takagi and M. Sugeno, "Fuzzy identification of systems andits application to modeling and control," IEEE Trans. Syst., Man,Cybern., vol.15, pp. 116-132, 1985

[2] J. C. Bezdek, Pattern Recognition with Fuzzy Objective FunctionAlgorithms, New York, Plenum, 1981

[3] D. E. Gustafson and W. C. Kessel, Fuzzy clustering with a fuzzycovariance matrix, in Proc. IEEE CDC, San Diego, CA, pp. 761-766, 1979

[4] D. S. Broomhead and D. Lowe, "multivariable fuctionalinterpolation and adaptive networks", Complex System, Vol.2,pp.321-323, 1988

[5] S. Chen, C.F. Cowan, and P.M. Grant, "Orthogonal least squares[8] S. Chen, C.F. Cowan, and P.M. Grant, "Orthogonal leastsquares learning algorithm for radial basis functions network,"IEEE Trans. Neural Networks, vol.2, pp.302-309, March 1991

[6] S. Chen, P.M. Grant and C.F.N. Cowan, "Orthogonal least-squares algorithm for training multi-output radial basis functionnetworks," IEE proceedings-F, vol.139, no.6, pp.378-384,December 1992

[7] 0. Ciftcioglu, "Enhanced Multivariable TS Fuzzy Modeling withMultivariable Fuzzy Sets without Decomposition" Proc. The 9`hWorld Multiconference on Systemics, Cybernetics andInformatics, July 10-13, 2005, Orlando, Florida, USA

[8] M Hanss, " Identification of enhanced fuzzy models with specialmembership functions and fuzzy rule bases", Engineering Appl.ofArtificial Intelligence, 12(1999), pp. 309-319

[9] 0. Ciftcioglu, "Identification of Enhanced Fuzzy Models withMultidimensional Fuzzy Membership Functions", Proc. FUZZ-IEEE 2005, May 22-25, 2005, Reno, Nevada, USA

[10]J. Park and I. W. Sandberg, "Universal Approximation UsingRadial Basis Function Networks", Neural Computations, vol.3,pp.246-257, 1991

155