AUTOMATED TROUBLESHOOTING OF MOBILE NETWORKS USING BAYESIAN NETWORKS

R.Barco1 , R.Guerrero2, G.Hylander2, L.Nielsen3, M.Partanen2, S.Patel41Dpt. Ingeniería de Comunicaciones. Universidad de Málaga.

2Nokia Networks (Málaga, Spain)3Nokia Networks (Aalborg, Denmark)

4Nokia Networks (Cambridge, UK)C/Severo Ochoa s/n. Edif.Inst. Universitarios, Pl.3, Parque Tecnológico de Anadalucía, Málaga (Spain)

Abstract

In the current telecommunication scenarios operatorshave to cope with fast technological changes whileincreasing operational efficiency, i.e. diminishingoperational expenditures and, at the same time,maximising performance of the networks. In this paperwe present an automated troubleshooting tool forcellular networks, based on Bayesian networks, whichwill contribute to improve operational efficiency. Wepropose some Bayesian models for diagnosis in mobilenetworks and we present a troubleshooting tool, whichuses those models to diagnose the cause of problems. Aknowledge acquisition tool is also presented, whichconverts the knowledge of troubleshooting experts intoBayesian models by means of a friendly user interface.The models and tools have been tested in real mobilenetworks and some results and conclusions are alsooutlined in this paper.

Key Words: Troubleshooting, diagnosis, Bayesiannetworks, mobile

1. Introduction

Mobile networks will face deep changes in the comingyears, due to the introduction of new technologies, newservices, and increased number of subscribers. In thepast, operators managed to cope with those changes andthe network growth by increasing their personnel.However, this is not a feasible strategy anymore.Furthermore, current expenditures for most operatorsare mainly focused on operation and maintenance, morethan on customer service or on investing in newequipment. Therefore, a viable option to maintainnetwork quality with the existing workforce, whilstintegrating new technology at the same time, is toincrease the level of automation.

Troubleshooting is part of the network operation andmaintenance process. If a cell is temporarily non-operational due to a fault, probably neighbouring cellswill also be affected and the final result will be that thewhole network performance will be decreased.Therefore, it is crucial to ensure that cells are rapidlybrought back into operation.

Currently, troubleshooting is mainly a manualprocess, in which the person looking into the reasons forthe problem has to carry out a series of checks in orderto establish the causes of the problem. In this process,several applications and databases have to be queried inorder to analyse performance data, cell configurationand hardware alarms. The growing size of cellularnetworks, together with the increasing complexity of thenetwork elements, creates a need for automatedtroubleshooting. In order to fulfil this requirement, wehave developed a prototype troubleshooting method andtool, which is currently undergoing trials and testing.

The troubleshooting tool automatically collects therequired information from the data sources and reasonswith this information related to the faulty cell in order togenerate a diagnosis of the cause(s) of the problem(s).

Thanks to the automated troubleshooting tool, highlyexperienced staff can be released from mundane dailytroubleshooting tasks and can concentrate on otheraspects of network optimisation, thus increasingnetwork performance. One additional benefit is thatknowledge from different experts can be stored withinthe tool, therefore making expert troubleshootingknowledge available to the business at all times.

In this paper we present an application of artificialintelligence to automated troubleshooting of cellularnetworks. First, we present the current manualtroubleshooting procedures and we introduce therequired elements for automatic troubleshooting. Theproposed solution for automated troubleshooting isbased on Bayesian networks, addressed in Section 3.

In Section 4 we propose certain types of Bayesianmodels with certain structural properties. The mainreason for investigating several structures is a trade-offbetween diagnosis accuracy and model complexity.

We have developed two prototypes: A troubleshootingtool, which is in charge of diagnosing the most probablecause of problems based on automatically collectedevidences and Bayesian models; and a knowledgeacquisition tool, which converts the knowledge fromtroubleshooting experts into Bayesian models (whichare used by the troubleshooting tool).

Both tools have been tested in real cellular networksand the results of the trials are presented in Section 7.

Finally, Section 8 summarises our contributions anddiscusses future work.

2. Operational Scenar io

2.1. Manual troubleshooting

The current scenario for troubleshooting in mostoperators' networks is shown in Fig.1. In the figure, themain elements involved in troubleshooting can beobserved:

� The reactive fault management team is responsiblefor dealing with alarms generated on the network. Theyfilter the important alarms and raise troubletickets,which reflect the problem status. Sometimes, they solvethe problem themselves and sometimes they leave thetroubleticket for further investigation.� The proactive fault management team finds poorlyperforming cells, based mainly on short-term statistics.This team often uses scripts to generate a list of "worstperforming BTSs" and takes it as a starting point fortheir work. Then, they look further into troubleticketsraised by the reactive team and raise new troubletickets.They can solve parameter related problems, but theywill need to involve field engineers for problems relatedto HW on the site.� Field engineers travel to the BTS sites and fix HWproblems and any other problem requiring on-sitepersonnel. The field engineers receive a new daily planevery morning containing the list of sites they mustvisit.� Finally, a minor part of troubletickets are raised bytechnical support teams who may receive customercomplaints from call centers, management, engineeringstaff, etc.

2.2. Automated troubleshooting

Automatic troubleshooting improves efficiency ofnetwork operation personnel, quickly identifying causesof problems and proposing solutions [1]. Furthermore, itcould be integrated with the management andtroubleticket system. The scenario for automatedtroubleshooting is shown in Fig.2.

Automated troubleshooting consists of three steps:

• Fault detection: automatic detection of badperforming cells based on performance indicators,alarms, etc.• Diagnosis or Cause identification: automaticreasoning mechanisms to identify the cause of theproblems and the best sequence of actions to solve them• Problem solving: executions of the action to solvethe problems

Hereafter when speaking about troubleshooting it willbe understood that we refer to cause identification,keeping in mind that troubleshooting has a wider scope.Fig.2 explains the automated troubleshooting procedure.First, the knowledge of the troubleshooters has to betransferred into a model for the system by using aknowledge acquisition tool (KAT). The Fault Detection(FD) subsystem indicates which are the cells withproblems and the kind of problem. The Troubleshooting

Fig.1. Manual troubleshooting

Fig.2. Automated troubleshooting

Troubleticket

Statistics

Fix HW problem(and close ticket)

Fix parameterrelated problems(and close ticket)

Field engineer'sdaily plan

Alarms

Radio Network System

Proactive fault managements team� Looking into statistics / performance� Receiving top-10 worst cells

Reactive fault managements team� Handling alarms

Field engineers

Technical support� Customer complaints

�����

User Observations

Target cell

� ���

Expert Knowledge

Action

Troubleticket

Statistics

Planning data

Cause

Alarms

Tool (TST) chooses the proper model and makes thereasoning based on user observations (e.g. antennadown tilt), configuration data (e.g. frequency reusesetup), statistics from databases (e.g. measuredinterference levels) and hardware alarms. The output ofthe troubleshooting tool is a list of possible causes witha probability associated to each cause. The tool alsorecommends the most cost efficient action to solve theproblem. E.g. even if a HW problem is a more probablethan a configuration problem, the troubleshooting toolwill recommend to change a parameter or to reset theequipment before sending someone to the site in orderto check if there is a HW problem. This is due to thefact that the first option is less expensive (time/cost)than sending someone to the site.

3. Bayesian Networks

An expert system simulates the human way of reasoningto infer conclusions from some available information.The automated troubleshooting tool is a decisionsupport system, which is an expert system that ratherthan trying to completely replace experts, providessupport for both experienced and less experiencedpersonnel. Several expert systems techniques have beendeveloped over the last decades, the simpler one beingrule-based expert systems [2].

The chosen technique for troubleshooting mobilenetworks has been Bayesian networks [3], which hasseveral advantages when compared to rule-basedsystems. Bayesian networks are probabilisticrepresentations for uncertain relations, which have beensuccessfully applied to real-world problems, asdiagnosis of medical diseases [4] and troubleshooting ofprinters [5].

In a Bayesian network, the domain is modelled bymeans of nodes or variables connected with arrows,which represent causal relations between the nodes. Thevariables can be continuous or discrete, having anumber of exclusive states. Probabilities areincorporated to the model as the "strength" of theconnecting arrows.

One important advantage of Bayesian networks is that ithas been proved that they are superior to othertechniques when dealing with uncertainty withindomains. Troubleshooting is an area in which Bayesiannetworks fit perfectly, as the relation between possiblecauses of problems and their corresponding symptomsare not deterministic.

There are known algorithms to efficiently inferknowledge from evidences, i.e. knowing the state ofsome variables, obtaining the probability of each stateof other unobserved variables. In that way, knowingsome symptoms of a given disease or cause of problemsin a mobile network, it is possible to deduce theprobability of the possible diseases or causes.

A bottleneck in Bayesian networks is knowledgeacquisition, i.e. converting the domain knowledge of

experts into Bayesian models. The structure of theBayesian network must be defined, i.e. the relationsbetween the nodes, and even if some simplifications aremade about the structure, still the number ofprobabilities that has to be specified is usually high.Furthermore, experts in troubleshooting normally do notknow how to build a Bayesian model directly. Theresults of the automated troubleshooting tool depend onhow well it was "taught" by the experts. Therefore,having a semi-automated knowledge acquisition tool tohelp the user seems to be an essential requirement.

Another key aspect of Bayesian networks is that theycan learn from experience, e.g. from a database withprevious cases, and continuously adapt to changes in thedomain. In that case, it is not needed that expertsspecify all probabilities with a high accuracy.

4. Models for mobile networks

A Bayesian model that can be applied to diagnosis inmobile networks is shown in Fig.3. It consists of threetypes of nodes:

� Causes: they represent the possible faults that maybe causing problems in the network (e.g. HW problem,interference in downlink, etc.)� Symptoms: they represent manifestations of thecauses (e.g. signal level decreased, increase number ofHOs, etc.)� Conditions: they represent factors that can have animpact on the causes (e.g. cell density: load problemsare more likely in densely populated ) or on somesymptoms (e.g. the average number of HOs is alsodifferent depending on the cell density)

In the example in Fig.3 there are two conditions(Frequency reuse, Cell density), which have an impacton the causes (Interference in DL, Coverage) and onsome symptoms (Downlink Quality HOs, UplinkQuality HOs). Furthermore, some symptoms may becombination of other symptoms (e.g. Downlink orUplink Quality HOs).

Once the structure of the model is defined, the states ofthe nodes should be specified and the probability tablesshould be filled in. When a node has several parents thisbecomes a cumbersome task because probabilities foreach combination of the parent nodes have to be set. In

Fig.3. Example of Bayesian network for troubleshooting

Interferencein DL

Coverage

Interferencelevel RX level

Freq.reuse Cell density

InterferenceHO

DL QualityHO

DL or ULQuality HO

UL QualityHO

order to simplify the knowledge acquisition two modelsare proposed, which consider some assumptions inorder to reduce the size of the probability tables: naïvemodel and causal independence models.

The naïve model shown in Fig. 4 has been extensivelyused in many diagnostic systems. When this model wasused in the medical domain, the parent node representeda set of alternative diseases and the children werepotential symptoms of the diseases. In the case oftroubleshooting mobile networks, the parent stands forthe possible causes of problems in the network, whereasthe children are the symptoms and conditions. Thenaïve model has some implicit restrictions: first, singlefault assumption, i.e. it supposes that only one cause ispresent at the same time and, second, the children(symptoms and conditions) are considered to beindependent given that the cause is known.

Causal Independence models overcome the limitationsof the naïve model [6]. One particular model is Noisy-OR [7], which assumes that each cause Ci will bringabout a related symptom Si to happen unless an"inhibitor" prevents it. While Noisy-Or requires that thecauses and symptoms are binary variables, other causalindependence models (e.h. Noisy-MAX, Noisy-ADD,etc.) allows any number of states.

The use of causal independence leads to simplificationsin probability assessment and inference. For example, inFig.5, if the probability table of symptom S is builtusing causal independence assumptions, the number ofprobabilities to be assessed is linear in relation to thenumber of parents, instead of exponential.

5. Troubleshooting Tool

An automated troubleshooting tool (TST) has beendeveloped in order to validate the previous concepts.The tool reasons in order to diagnose the cause of theproblem(s) in the network, e.g. high number of droppedcalls. The conclusions of the TST are based on thefollowing data, which should be introduced to thesystem before starting the automated troubleshooting:

� Fault cases, e.g. congestion, high drop call-rate, etc.,which determines the Bayesian model to use.

� Cells to be analysed (e.g. top-10 bad performingcells according to statistics collected). Normally thiswill be automatically fed by the Fault Detection system.� Dates of the analysis together with the averagingmethod used to calculate the performance statistics(busy hour, 24 hours data…).

When the previous information is entered into the tool(Fig.6), the user will be asked those data that cannot beautomatically collected from the data sources (e.g.weather condition, cell density…).

Next step is to run the analysis or schedule it, forexample run TST during early hours of the morning sothat analysis is ready when engineers come in themorning.

During the analysis the TST collects data from allsources specified in the model (databases, alarms, etc.).When the data collection finishes, the TST performs thereasoning, based on the selected Bayesian model, andusing an inference engine to calculate the probability ofeach possible cause being the one causing theproblem(s).

The conclusions of the troubleshooting tool aredisplayed when the analysis is finished (Fig.7). TheTST presents, for each cell, the list of possible causes ofthe problem ranked by their efficiency (function of theprobability and the cost of the action required to solvethe problem).

Fig.6. Selection of Target Cells and Dates Options

Fig.5. Bayes model where causal independence canbe applied to build the probability table of node SFig.4. Naïve Bayes model

S

C2 C3 C4 C5C1Causes

Symptom1 Symptom2 Condition1 ConditionM… …

All the collected evidence related to symptoms can bedisplayed or saved locally in text format. The user canalso save his own feedback on the problem causetogether with the results of the analysis. Thisinformation can be utilised by the expert trouble-shooterto verify if the solution provided by the tool was theright one.

6. Knowledge Acquisition Tool

The performance of the troubleshooting tool willdepend on the knowledge of the system, i.e. theknowledge of the troubleshooter experts should beprecisely transferred to the tool in the shape of Bayesiannetworks. Normally a knowledge engineer is required toconvert the knowledge of the expert into Bayesianmodels, which should include determining theimportant variables to consider, states of thesevariables, relations among the variables, probabilities,etc.

The Knowledge Acquisition Tool (KAT) plays the roleof a knowledge engineer by guiding the troubleshooterin creating the network. In order to simplify theknowledge acquisition, some assumptions about thestructure of the model are considered (see modelsproposed in section 4). These assumptions will allowthe user to insert a minimal quantity of information thatlater on will be used by KAT as a seed to automaticallycomplete all the information needed to build theBayesian network.

The steps for the creation of a network are:

� Definition of causes, symptoms, conditions andactions. They can be reused in different networks. In thecase of symptoms and conditions the user has toestablish which script (e.g. look up in a database) willbe used to collect the data. KAT provides a wide rangeof scripts grouped by category e.g handovers,interference, congestion, etc.� Selection of a fault case e.g high drop call rate, andsituations that can cause it e.g. Abis problems, badcoverage, etc.� Selection of symptoms and conditions for eachcause in the model.

� Specification of probabilities for causes, symptomsand conditions, including probability of each linkamong them (conditional probabilities).

KAT will guide the expert through the previous stepsand will inform him if there are any inconsistencies inthe data (Fig.8). Based on the previous information,KAT will automatically create a Bayesian model (naïveor conditional independence model) that will be used bythe Troubleshooting Tool.

When dealing with troubleshooting experts, it has beenrealised that specifying probabilities for a givenstructure is quite difficult and different experts canprovide completely different values, which can lead toinaccurate results. Normally, operators have systemdatabases containing the history of most variables in thenetwork, which can be used to train the Bayesiannetwork and obtain the probabilities based on thoseprevious cases.

7. Results

The development of TST has been carried out in co-operation with an operator. Testing the accuracy of themodel, features of Bayesian networks and thefunctionalities of the prototype have been performedwithin a real network environment. There was directinput from expert troubleshooters into the developmentfor both the modelling and the tools functionalities. Thefault scenario selected for testing and modelling is builtupon troubleshooting cells having high drop call rate.Accuracy and hitrate of solutions are determined fromperforming manual troubleshooting and comparing withresults from TST for the same cells and conditionsaffecting those cells. An iterative approach to improvingthe accuracy and hitrate was regarded suitable: the mainimprovements in the TST hitrate have come from directfine tuning of the model by the expert troubleshooters.

The TST's performance was derived from recordedcases where the manual analysis was performed and asolution for the majority of the problem cases found.There existed cases where commonly, site resets were

Fig.7. Results of the analysis

Fig.8. KAT main window

performed to alleviate the problem and this rendered in-depth investigation impossible as normal service wasresumed. The other problem cases were fed into TSTand its results stored for analysis and comparison. Theresulting hitrate as shown in Figure 9 showsperformance %'s over two separate periods, a fewweeks apart during testing in 2001.

The results showed that in first tests the output wasoften a false diagnosis, but the correct cause usuallyscored quite well, and the highest ranked cause usuallymade some sense based on the inputs. We also observedthat the domain expert could make little changes basedon the first trials to make the diagnosis output muchmore accurate. In a few weeks we got a "hitrate" ofaround 60-70% on the top-ranked cause.

8. Conclusions and future

This paper presented an application for automatedtroubleshooting of mobile networks based on Bayesiannetworks. Appropriate models for the domain werefound after an iterative process of interaction withtroubleshooting experts. Troubleshooting based onBayesian networks was proven to be much moreefficient than troubleshooting based on rules, due to theintrinsic capacity of Bayesian network of dealing withuncertainty.

A knowledge acquisition tool has also been developed,which helps troubleshooting experts to convert theirknowledge into Bayesian models. It has proven to bevery useful when dealing with experts in existingoperators' networks.

Trials have been carried out in real networks, showingpromising results. The obtained hit rate is similar to theone obtained if experts manually diagnose the problems.Moreover, the response time of the tool is much lessthan the time required by a human expert. Trials are stillongoing, fine tuning the model when new cases areanalysed. The performance improvement is quite highevery time the model is modified, which makes us thinkthat even better results will be obtained in the future.

Furthermore, there are future research areas that willcontribute to further improve the results. Learning willmake the model not so much dependent on the experts'accuracy and will lift the workload associated to createand tune the models. Another research area is related todiscretization of continuous variables, i.e. defining theadequate number of states and thresholds for each stateof a symptom that is continuous by nature.

9. Acknowledgements

This work has been performed as part of the co-operation agreement between Nokia and the Universityof Malaga. This agreement is partially supported by theProgram to promote technical research (Programa deFomento de la Investigación Técnica, PROFIT) of theSpanish Ministry of Science and Technology.

The authors would like to thank Orange andSonofon for fruitful discussions and feedback. We alsowould like to thank Dr.Finn Jensen at University ofAalborg for his valuable advice.

References

[1] Halonen, T., J.Melero, J.Romero, GSM, GPRS &EDGE Performance: Evolution towards 3G UMTS(Wiley, 2002)[2] Russel, S., P.Norvig. Artificial Intelligence - amodern approach (Prentice Hall, 1995)[3] Jensen F., Bayesian networks and decision graphs(Springer, 2001)[4] J.E.Desmedt (Ed.), Computer-AidedElectromyograpgy and Expert Systems (ElsevierScience Publishers, Amsterdam 1990)[5] Heckerman, D., J.Breese, and K.Rommelse.Decision-theoretic troubleshooting. Communication ofthe ACM, 1995, 38 (3), 49-56[6] Heckerman, D., J.Breese. Causal Independence forprobability assessment and interference using BayesianNetworks, IEEE, Systems, Man, and Cybernetics, 1995[7] Kim, J., and J.Pearl. A computational model forcasual and diagnostic reasoning in inference engines. 8th

International Joint Conference on ArtificialIntelligence, 1983, 190-193

Fig.9. TST hit rate

���������

�

� ��

���� ������������� ��������� ������������� � ������� ������������� � ���

���� ���"!#�$�&%����'! (*),+�- �.�0/ � ������1! 2$��- ! 23��4���

� ! �5��! �76 � �4- 85�9+4��6 � ��- ��!�);:<�'6 ��- =$6 �>���� ��� ?�+�- �.�0/ ����� � - @A��6 BC���� ���>),+�- �.�0/ � ��� �

D,E�F