TM1 -2:30

F’usion-based Sensor Fault Detection Sangwook Park and C. S. George Lee *

School of Electrical Engineering, Purdue University, West Lafayette, IN 47907

Abstract

A Fusion-based Fault Detection and Di- agnosis (FFDD) system is introduced for a multisensor system where the fault detection process is divided into three staejes and the Uncertainy-Reductive- Fusion Technique is used to detect for the possible sensor failures. A fuzzy rule- based diagnosis is performed in order to analyze and t o t ry to correct for the sig- naled faults.

1 Introduction

One characteristic of an “intelligent” sys- tem is its ability to self-adapt t o variations in the outside environment as well as the internal changes occuring within the sys- tem. Thus, the robustness of an “intelli- gent” system can be measured in terms of the sensitivity and adaptability t o such in- ternal and external variations. If a fault is defined as “a nonpermitted deviation of a characteristic property which leads t o the inability to fulfill the intended purpose” [4] , then internal parameter variations, noise degradation, external disturbances, and unexpected environmental changes which disturb the system from its normal state of operation are all considered to be faults. Therefore, Fault Detection and Identifica- tion (FDI) is a fundamental and integral part of any “intelligent” system. This pa- per focuses on detecting sensor failures in a multisensor system where multiple re- dundant sensors, identical or disparate, are used to measure and determine the same world feature or the same internal state of a system. The purpose of using multiple sensors is t o increase the reliabil- ity and the flexibility of the system.

‘This work was supported in part by the National Sci- ence Foundation under Grant CDR 8803017 to the En- gineering Research Center for Intelligent Manufacturing Systems.

Frank 21 and Isermann [5 ] provide a sur-

timation and parameter estimation tech- niques, respectively. More recently AI techniques such as the use of shallow and deep knowledge-based reasoning mecha- nisms have been investigated for the FDI problem where fault propagation trees were synthesized and failure diagnostics were performed using expert systems. Tzafestas [13] provides a survey of some AI approaches to the FDI. We propose the Fusion-based Fault Detection and Di- agnosis (FFDD) system where the sensory information is modeled using Fuzzy num- bers, F-numbers, and the Uncertainty- Reductive-Fusion Technique (URFT) [ll] is used to detect for the possible sensor failures in the system. The sensor faults are defined t o be unfusible sensory infor- mation where the multisensor system is initially calibrated t o provide fusible in- formation. The fault detection process is divided into three stages and the output of the three detection stages are monitored for the possiblity of sensor failures. The objective of the FFDD is t o provide the most reliable information possible among the multiple competitive sensory informa- tion.

vey of t 6 e FDI systems using state es-

2 The Fusion Process

Let mi be the output of sensor i. Then the sensory information can be modeled us- ing a Triangular F-number with the mean- value equal to m; and the spread cr; used to represent the confidence level of the sen- sory information. The confidence level, c;, is defined to be the reciprocal of the in- herent uncertainty, U;, associated with the sensory data.

Higashi and Klir’s U-uncertainty measure [6] is used to measure the uncertainty U ;

0-7803-1 206-6/93/$3.00 91993 IEEE 156

associated with each F-number F;. 1

u(Fi) = U; = C(pj - pj+l) * logIApfl (2) j=1

where p j ( j = 1,2 , . . .,l)-is the ordered possi- bility distribution of F; and APJ is the level set at p ’ .

Let the sheafof F-numbers be the set of N input F-numbers F1, j 2 , . . ., PN which are the outputs of the N sensors. The Uncertainty-Reductive-Fusion Technique, G(-), consists of two independent fusion subprocesses gm( . ) , the fusion of mean- values ml, m2, ... , ”; and gu(.) (other- wise known as uncertainty fusion), the fu- sion of the uncertainty levels U;, 142, . . . , UN associated with each input F-number.

The mean-value fusion process is sim- ply a Linear Opinion Pool with confidence weighting and is given by,

n

gm(m1, m2, * 7 ~ I V ) = m~ = C wi * mi (3) i=l

where

by,

~ ~ ( u I , u ~ , - * . , u N ) = uc = Xumin + (1 - X)umaz

(4)

w; = 1 and 20; = +. C,=l cl

The uncertainty fusion process is given

where Umaz = C;U(F;) and Umin = U T e j ( f ’ ~ , P2,. . . , &) which is determined from the Principle of Maximum Confirma- tion [ll]. X is defined to be the degree of confirmat ion,

1 X = -U(?r&) * ?r2(z) * . . . * ? r N ( X ) ) ( 5 )

Umin

where * is the algebraic product t-norm and ?r;(z) is the possibility distribution as- sociated with each input F-number.

The sheaf of input F-numbers are said to be “fusible” if and only if,

U C < min[u1, U2, . . . , U N ]

3 Fusion-based Fault Detec- tion and Diagnosis System

There are many types of sensor faults which can arise for multisensor systems. The list of sensor failure states consid- ered in this paper are: failure of the hard- ware of the physical sensor (SF), failure of the whole mulitsensor system (SSF),

noise (N), global environmental variations (GEV) such as changes in the lighting con- dition, and local environmental variations (LEV) such as motion of the object in the world. The goal of the fault detection problem is to make sure that a fault or fail- ure is always detected when a fault actu- ally occurs and t o never send off an alarm when there is no fault. Since the first goal is more detrimental to the system than the second goal, the fault detection process is designed to be very sensitive to all types of failures and an “intelligent” and efficient fault diagnosis process is designed in or- der to make the correct decisions in case of an alarm. The FFDD is designed to be robust for any given failure state of the system and to minimize the propagation of faults from one stage to the next.

3.1 Fault Detection

The faults are detected in three stages: stage 1 is the multisensor fusion process, stage 2 is the temporal fusion process, and stage 3 is the diagnostic fusion process (see Figure 1). The subdivision of the fault de- tection process into three separate stages is to minimize fault propagation and to aid the fault diagnosis process by provid- ing more detailed information about the sensory information and the state of the system. Initially, the outputs of the Ai re- dundant sensors are fused together using the URFT to determine the single repre- sentative group consensus. Since we are dealing with competitive sensory informa- tion, the system is calibrated in order to provide fusible results and any unfusible information at any one of the three detec- tion stages would signal the possibility of a sensor failure in the system, particularly the sensor which produced the unfusible sensory information.

The temporal fusion process (stage 2) is used to check for time consistency with the prior information stored in the system database. The URFT is again used for the fusion of time-based information and tem- poral fusion is mainly used to detect object motion. If the object in the world moves or the sensor moves, then the multisensor fusion process (stage 1) may not produce a conflict but the temporal fusion process will. The fusion is performed between the consensus result of the multisensor fusion process, Fc , and the output of the FFDD for one prior time instance, Fct-l .

The last fault detection stage is the di- agnostics fusion stage (stage 3). For this stage, the information about the nominal

157

I

stage 3

Figure 1: Fault detection process

state, F*, of the system is needed and this is provided by the system database where the last no-failure consensus result of the FFDD is defined to be the nominal state of the system because the system is dy- namically changing and no models of the system are assumed. The elements in the conflict-list, if t here are any, of the mult isen- sor fusion process are clustered into fusible groups GI, Ga,. . . , G, using the URFT and the consensus results of each of the fusible groups are denoted as FG, for j = 1 ,2 , . . ., T .

Each p~~ is fused with the nominal infor- mation E* and this process is called the diagnostic fusion stage. Note that since the URFT is used for clustering, either the temporal fusion process will provide a fusible output or the diagnostic fusion process will provide a fusible output or neither. Also there will be a t most only one fusible output from the diagnostic fu- sion process among the T fusible groups. A multiplexer is used to choose between the results of the temporal fusion process or the diagnostic fusion processes in order to produce the final output, Fct, of the FFDD.

The FFDD system is designed to be very sensitive to the occurrence of faults in the system. However, in order not t o unnec- essarily stop the operation of the system whenever a fault is detected, an efficient fault diagnosis process is needed. The sys- tem is only stopped and undergoes a full diagnostics check when no fusible results are provided by all three stages of fault detection - which is rare. So both goals of the FDI systems are realized; the sys- tem is very sensitive to faults because the URFT is designed to be very restrictive in

its fusion process and if the fault diagnosis can be run in parallel (or off-line) then the system will keep on operating at nominal state until a SSF fault is diagnosed. The redundany provided by the multiple com- petitive sensory information is the basis for the system’s fault-tolerance.

3.2 Rule-based Fuzzy Diagnosis using

Figure 2 shows the term set for the reasoning process and Table 1 shows the full fuzzy rule-base for the fault- diagnostics logic. Assume that all the in- put F-numbers have the same uncertainty level (spread of 1.0) and assume that there are four sensors, i.e. N equals 4. The in- novation A u c determines the number of F-numbers fused in the multisensor fusion process (stage 1 of the fault detection). The linguistic variable “very small” (VS) is used for the innovation A u c to mean that all the input F-numbers were fusible and the range of values for the linguistic variable “very small” is [ O . O , 0.3041. Siin- ilarly, “small” (S) is used to mean that 3 out of the 4 input F-numbers were fusible with range of values [0.304 , 0.7301; “large” (L) is used to mean that 2 F-numbers were fusible with range of values [0.730 , 1.3271; and “very large” (VL) is used t o mean that all four F-numbers were unfusible and one of the input F-number was chosen as the output of the multisensor fusion process (range of values > 1.397).

innovation Anzc determines whether the temporal fusion process was successful or not and the innovation Am, determines whether the information from sensor i (the sensor in the conflict-list and the sensor un- der diagnostic check) was fusible or not with the nominal value, m*. Both inno- vations, A m c and Ami, are differences in the mean-values, Amc = nzc - m* Anz; = mit - m*. The linguistic variables used for the difference in mean-values were: “very small” (VS) meaning that the two input F-numbers were fusible with range of val- ues [0 , 0.51; “small” (S) meaning that the two F-numbers had some overlapping in- formation (range [0.5 , 2.01); “large” (L) meaning that there were no overlapping information (range [2.0 , 5.01); and “very large” (VL) meaning that the F-numbers were very far away (range > 5.0). Usually, the cases for “large” and “very large” are identical unless otherwise stated.

Once a sensor output is signaled to be unfusible, the particular sensor is tem-

Innovations

The

1.0

0.8

0.6

0.4

0.2

0.0

DitfereKs in the mean-vdues

1 .o

0 8

06

0 4

0 2

0 0 1 5 2 0 2 5 2 8 32

[)lfleronm in the uncartainty vdues

Figure 2: Term set of the reasoning process

porarily disconnected from the multisen- sor system and put under diagnosis. This is the advantage of using multiple redun- dant sensors because the overall system can continue t o operate even though a few of the sensors are put under diagnosis and removed from the system. The first step in the diagnosis process is to t ry to cor- rect for the fault by performing some self- correcting diagnostics. The sensor retakes its measurement and reprocesses the new information to see if the fault has been cor- rected. If so, then the original error was considered t o be due t o noise and the sen- sor is placed back into the system. If not, then some of processing methods or pa- rameters involved in the processing of the sensory information is changed t o see if the fault was due t o processing limitations or errors. If correctable, fine but if not, then the sensor is diagnosed t o be faulty and must be either replaced or fixed.

The decision parameters Auc , Amc, Am;, and whether the fault was cor- rectable or not (y/n), defines the fuzzy rule-base for the fault diagnosis. The con- sequence of the rules provide a degree of possibility for each of the five types of faults: SSF, LEV, SF, N, GEV. The de- gree of possibility is divided into four lev- els: very possible (VP), possible (P), not- so possible (NSP), and not possible (NP). The fault with the highest possibility was chosen as the result of the diagnosis, where

V P > P > NSP > NP.

Table 1: Rule-base for Fault Diagnosis

VL

LEV NI-' P V P NI-' N P N P N P VlJ- P V P P vp P V P P V P P N S P N S P N S P N S P Y N S P P N S P P N S P P N S P P N S P w N P N S P N P Y N P P N P msp N S P N S P N S P N S P N S P -

SF -NF- N P N P mp- P N S P V P msp N S P N S P V P w N S P N S P V P N S P V P N S P V P N S P V P w N S P N S P V P w N S P N S P V P N S P V P w V P N S P V P -mP- N S P N S P V P msFr N S P N S P V P N S P V P

V P V P T P P

gp 1:: V P P

4 Simulation Results For simulation purposes, virtual sensors were used rather than real physical sensors because faults are less costly and easier to simulate for virtual sensors, which are just software models. The multisensor system was simulated by assuming that a virtual sensor was simply a different computer vi- sion processing system working with the same input raw image. This assumptions is justifiable since our formulation is inde- pendent of the particular types of sensors and all that is needed are different infor- mation sources providing competitive in- formation. A CCD camera connected to a Datacube Vision system was the physi- cal vision system used. The images were taken and stored in a Sun workstation to

159

be processed off-line; note that the fusion process itself can be implemented real- time but the computer vision processing can not be.

The task we chose t o test the FFDD was a very simple one where the virtual sen- fors were used to locate an object in an image. The main reason for using simple tasks and simple processing tools was to test the virtual faults for these basic tasks and the added complexity of more compli- cated systems is not necessary and might even cloud the real issues involved. The problems involved with fault detection and diagnosis is clearly illustrated even for this simple task and formulation. Our work is currently being extended to be imple- mented for a real multisensor system.

Figure 3(a) is the control image used. The fault conditions were simulated by manually moving the object and tak- ing another picture for processing (LEV fault, Figure 3(b)); by manually changing the lighting condition (GEV fault, Figure 3(c)); and by adding Gaussian noise to the image (N fault, Figure 3 (d)(e)). Sen- sor faults (SF) were defined to be failures due to processing limitations of each vir- tual sensor and SF was simulated by par- tially occluding the object (Figure 3(f)). The whole sensor system was defined to be unoperational when a SSF arose and this happened only when all the virtual sensors provided unsatisfactory results. Table 2 shows the fusion results for all the cases.

Sensor 1 - threshold image with value T I , run a connectivity algorithm to determine the largest object in the image, determine the center of the object using the maxi- mum and minimum pixel values (id), and finally associate a F-number with the re- sultant location information.

Sensor 2 - threshold image with value T2, run a connectivity algorithm to deter- mine the largest object in the image, per- form binary edge detection, window t he ima e to only include the region around the fargest object, convolve the windowed image using a corners mask, locate all the pixels where the corners mask fits, calcu- late the center of the object using these pixels, and finally associate a F-number with the resultant location information.

Sensor 3 - threshold image with value T3, run a connectivity algorithm to determine the largest object in the image, determine the center of the object using the center of mass algorithm, and finally associate a F-number with the resultant location in- formation.

Sensor 4 - perform edge detection, con-

(yj 3e x

y 31 (xl

Table 2: Results of the siniulation

206 470 56 310 361 310 i2.929) 240 275 269 248 2.52 248 (2.929 GEV 353 370 306 360 361 360 (2.929 255 238 221 248 2.52 248 (2.9291 SF .

(y) I 358 I 330 1 342 I 360 I 361 1 36Oi2.623j I

Figure 3: (a) Control image

volve the image using a circle mask, deter- mine the center of the object by locating the pixel with the ”best” fit, and finally associate a F-number with the resultant location information.

References R. N. Clark. “Instrument Fault De- tection”, IEEE Trans. on Aerospace a n d Electronac Syslems, AES-14, 110.3, 1978, pp.456-465

P. M. Rank. “Fault Daignosis in Dy- namic Systems via State Estimation: A Survey”, System Fault Dzagnostzcs, Relaabalziy and Related Knowledge-based Approaches : Vol- ume 1, Fault Dtagnositcs and Relzabzlzt (ed) S. Tzafestas, M. Singh, and G. Scknidt. D. Reidel Publ. Co., Netherlands, 1987.

P. Gmytrasiewicz, J. A. Hassberger and J. C. Lee. “Fault Tree Based Diagnostics using Fuzzy Logic”, IEEE Trans. PAMI, v01.12, no.11, 1990, pp.1115-1119.

Figure 3: (b) Moved object (LEV fault)

180

Figure 3: (c) SNR 10 Gaussian-noise added im- ages (No faults)

Figure 3: (d) SNR 2 Gaussian-noise added im- ages (SSF faults)

Figure 3: (e) Change in lighting condition (GEV fault)

Figure 3: (f) Object partially-occluded (SF fault)

[4] R. Isermann. Process Fault Detection Based on Modeling and Estimation Methods - A Survey, Automatica, v01.20, no.4, 1984, pp.387-404.

[5] R. Isermann. “Experiences with process fault detection methods via parameter es- timation”, System Fault Diagnostics, Relia- bilit y and Related Knowledge-based Approaches : Volume 1, Fault Diagnostics and Reliabil- ity, (ed) S. Tzafestas, M. Singh, and G. Schmidt. D. Reidel Publ. Co., Nether- lands, 1987.

[6] G. J. Klir and T. A. Folger. Fuzzy Sets, Un- certainty, and Information , Prentice-Hall, New Jersey, 1988.

[7] R. Krzysztofowitz and D. Lond. ”Fusion of Detection Probabilites and Compari- son of Multisensor Systems”, IEEE Trans. on Systems, Man, and Cybernetics , v01.20, 110.3, 1990, pp.665-677.

Mech-

Trans. on Reliability, vo1.40, no.5, 1991,

[9] J. von Neumann. “Probabilistic Logics and the Synthesis of Reliable Organisms from Unreliable Components”, Automata Studies, (ed) C. E. Shannon and J. Mc- Carthy, Princeton University Press, New Jersey, pp.43-98, 1956.

[lo] B. Parhami. “Voting Networks”, IEEE Trans. on Reliability, ~01.40, no.3, 1991,

I l l ] S . W. Park and C. S. George Lee. “Uncer- tainty Fusion of Sensory Information Us- ing Fuzzy Numbers”, Proceedings of the 5th IFSA World Congress, Seoul, Korea, July, 1993.

[12] R. Swinburne. An Introduction to Cotifirma- iion Theory , Methuen and Co. Ltd., Lon- don, UK, 1973.

(131 S . G. Tzafestas. “A Look at the Knowledge-based Approaches to System Fault Diagnosis and Supervisory Con- trol “, System Fault Diagnostics, Reliabiliiy and Related Knowledge-based Approaches : Vol- ume 2, Knowledge- based and Fault- Tolerant Techniques, (ed) S. Tzafestas, M. Singh, and G. Schmidt. D. Reidel Publ. Co., Netherlands, 1987.

(141 A. S. Willsky. “A Survey of Design Meth- ods for Failure Detection in Dynamic Sys- tems”, Automatica, v01.12, 1976, pp.601- 611.

[15] C.C. Yu and C. Lee. “Fault Diagnosis Based on Qualitative/Quantitative Pro- cess Knowledge”, AIChE Journal, vo1.37, no.4, 1991, pp.617-628.

[SI A. Kumar and K. Malik. “Votin anisms in Distributed Systems B , IEEE

pp.593-600.

pp.380-394.

161