Abstract—We propose a method based on Hidden Markov

Models (HMM) for filtering out users who submit unfair ratings to an online reputation system. We assume that users are fair, positively biased, and negatively biased, and each user submits a number of ratings. A different HMM (A, B, π) is used to describe each of the three types of user, with the only difference in these HMMs being matrix B. For each user, we train an HMM based on the user's ratings, and subsequently the B matrices of the trained HMMs are clustered together into two groups: fair and unfair (both positively and negatively biased), which permits us to identify the group of fair users. The A matrices of the filtered fair users are then aggregated in order to estimate the A matrix of the quality of the service under study. Also, using the HMMs of the fair users, we can obtain the most probable current state of the quality of the service. Finally, we show that our filtering algorithm has a good accuracy when the number of unfair users is varied from 0% to 50%.

Index Terms— Hidden Markov Models, HMM, filtering unfair ratings, trust estimation, clustering.

I. INTRODUCTION NLINE reputation systems are commonly used in association with e-commerce websites through which users

buy services and products. These systems allow users to submit their feedback ratings for services and products that they have purchased so that to help future users to make a decision as to whether to buy a specific service (or product). In such feedback mechanisms, it is important to identify and remove users who submit deliberately unfair ratings in order to influence the overall rating of a service. Unreliable ratings can be categorized into unfair positive ratings and unfair negative ratings. Users who collude with another seller give unfair negative ratings to a competing seller in order to decrease its reputation. Likewise, users bribed by a seller submit unfair positive ratings for the seller in order to increase its overall rating. In this paper, we propose a method to filter the unfair users, be it positively or negatively biased, using a Hidden Markov Model (HMM).

There have been several papers in the literature that have addressed the issue of filtering out unfair ratings. These papers

can be logically organized into two groups depending on the method used. In the first group, the methods used are based on the majority rule, that is, ratings that deviate from the majority are classified as unfair. This rule obviously does not perform well when a lot of the ratings are unreliable. To alleviate this problem, [1] introduced two factors, namely, familiarity and consistency, in order to better identify unreliable ratings. [2] used statistical clustering to separate fair ratings from positively biased ratings. The ratings are clustered into two groups: a lower and a higher ratings clusters, and the fair rating are those in the lower cluster. In [3], the authors proposed a set of methods based on signal processing technique to detect unfair ratings. Specifically, they detect the time intervals in which dishonest ratings are highly likely present. This method may not work well if the unfair ratings are spread over the entire rating duration. Trust-in-raters is an important technique for detecting unfair ratings. In [4] the authors proposed a personalized approach for handling unfair raters in a centralized reputation system. Specifically, the system allows a buyer to estimate the reputation of raters based on their ratings for commonly rated sellers. In addition, the reputation of the sellers is also estimated using the ratings submitted by the raters. In [5], the authors proposed a scheme that detects collaborative unfair raters based on similarity in their rating behaviors. This is based on the notion proposed by the authors that the collaborative unfair users always work together for achieving the same goal. For better accuracy and avoiding sparse ratings, both signal processing and trust-in-rater techniques, were used. Finally, the Bayesian reputation scheme proposed by [6], filters out unfair ratings of an agent from fair ones, based on the notion that unfair raters have a different statistical pattern from fair ones. This scheme determines dynamically an upper and lower threshold and a rater is considered fair if its score is within the bounds. This approach is only effective when the significant majority of ratings are trustworthy.

In the second group of papers on filtering unreliable ratings, the proposed method is based on HMMs. In [7], the authors proposed an HMM trust model for autonomous agents in a multi-agent environment to predict the state of an agent. In [8], the authors proposed a method for HMMs using the particle swarm optimization procedure. They applied their method to predicting the next rating of a user, assuming only

Filtering Unfair Users: A Hidden Markov Model Approach

Bushra Anjum1, Manish Rajangam2, Harry Perros2, Wenjuan Fan3

1Computer Science Department, Missouri University of Science and Technology, USA 2Computer Science Department, North Carolina State University, USA

3School of Management, Hefei University of Technology, Hefei, Anhui, China

0

fair users. Similarly, in [9] the authors proposed an HMM model to predict the next outcome of an entity, a rating in our case.

We note that in the above papers [7], [8], and [9], the results obtained are limited and they are obtained by solving problem 2 for HMMs, see next section. In this paper, we assume users are fair, positively biased, and negatively biased, and each user submits a number of ratings. A different HMM (A, B, π) is used to describe each of the three types of user. As will be seen, the only difference in these HMMs is matrix B. For each user, we train an HMM based on the user's ratings. The B matrices of the resulting HMMs are subsequently clustered together into two groups, one for fair and another for unfair (both positively and negatively biased) users, which permits us to identify the fair users. The A matrices of the filtered fair users are then aggregated in order to estimate the A matrix of the quality of the service under study. Also, using the HMMs of the fair users, we obtain the most probable current state of the quality of the service. Finally, we show that our filtering algorithm has a good accuracy when the number of unfair users is varied from 0% to 50%.

HMMs are also used in [10] but for a different problem related to bootstrapping trust of a Web service. The observations sequence for a particular Web service is matched against pre-defined trust patterns, in order to assess the behavior of service. The pre-defined trust patterns are specifications of possible behaviors of Web services such as trusted, malicious, betraying, oscillating, and redemptive. Based on the matching result, an initial trust value is assigned to the Web service.

The paper is organized as follows. In the next section, we give a brief overview of the three problems associated with HMMs. In section III, we describe the proposed algorithm, and in section IV we give numerical results. The conclusions are given in section V.

II. A BRIEF REVIEW OF HIDDEN MARKOV MODELS A Hidden Markov Model is a Markov Chain with n states,

which are hidden from the user. When the Markov chain is in a state, it produces an observation with a given state-dependent probability distribution. There are M different observations. It is this sequence of observations that the user sees, and from which it is possible to estimate the parameters of the HMM. An HMM is defined by the A matrix which is the one-step transition matrix of the Markov chain, the B matrix (referred to as the event matrix) which contains the probability distribution of the observations likely to occur when the Markov chain is in a given state, and π, the vector of probabilities of the initial state of the Markov chain. The (A, B, π) are together represented by λ. In this paper, we will make use of the following notation:

A – The one-step transition matrix (N x N) aij -The (i,j)th element of A. It is the one-step probability

from state i to state j B – The event matrix (N x M) bi(k) – The (i,k)th element of B. It is the probability that

the kth value will be observed when the system is in state i

π - Initial Probability vector (Nx1) N - Number of hidden states M - Number of observations O - An observation sequence T - Length of the observation sequence Q - A sequence of hidden states traversed by the system λ = (A, B, π)

Three different, but related, problems have been defined for HMMs, as described briefly below.

Problem 1 - Forward Probability Computation: Given an observation sequence, compute the probability that it came from a given λ.

Let the system be in state i at time t. The probability of the system shifting to state j at t+1 is given by the one-step transition probability aij. The probability that of the M possible observed states, the kth one is observed given that the system shifted from state i to j is aijbj(k). Since the state i can be any one of the states from 1 to N, the probability of observing the kth value, given that the system shifts to state j at time t+1 is the sum of all probable paths to state j, i.e., 𝑎!"!

!!! 𝑏!(𝑘). Now, the only remaining unknown is the joint probability of

having observed the sequence from O1 through Ot and being in state i at time t, given λ. Representing this quantity by αt(i) and representing the kth observed value at time t+1 by Ot+1, we have:

𝛼!!! 𝑗 = 𝛼! 𝑖 𝑎!"

!

!!!

𝑏! 𝑂!!! , 𝑡 ∈ 1,𝑇 − 1 , 𝑗 ∈ [1,𝑁]

To start off this induction, the initialization step for calculating αt(i) is obtained as follows. The probability of the system being in state i at time 1 is given by πi and the probability of observing O1 is then given by bj(O1). Thus,

𝛼! 𝑖 = 𝜋!𝑏! 𝑂! , 𝑖 ∈ [1,𝑁] αT(i) obtained from the induction step represents the probability of observing the sequence from O1 through OT, and ending up in state i. Thus, the total probability of observing the sequence O1 through OT, given λ is

𝑃 𝑂|𝜆 = 𝛼! 𝑖!

!!!

Problem 2 - Backward Probability Computation: Given an observation sequence, compute the probable states the system passed through.

The quantity βt(i) is defined as the probability that the sequence of observations from Ot+1 through OT are observed starting at state i at time t for a given λ. It is calculated in the same way as αt(i), but in the backward direction. We have:

𝛽! 𝑖 = 𝛽!!! 𝑗 𝑎!"

!

!!!

𝑏! 𝑂!!! , 𝑡 = 𝑇 − 1,… , 1

For t = T we have βT(i)=1, i = 1, 2, …, N. To calculate the probability that the system was in state i at time t given O and λ, we observe that αt(i) accounts for the observation sequence from O1 through Ot and βt(i) accounts for Ot+1 through OT, and

both account for the state i at time t. So the required probability is given by αt(i)βt(i). Introducing the normalizing factor from problem 1, P(O|λ), we have

𝛾! 𝑖 =𝛼! 𝑖 𝛽! 𝑖𝑃(𝑂|𝜆)

At each time t, the state with the highest γ is the most probable state at time t. A better way to obtain the most probable path of states Q that give rise to an observation sequence O, is to use dynamic programming, as described in [11].

Problem 3 - Matrix Estimation: Given an observation sequence O, compute the most probable λ.

The term aij can be calculated as the ration of the number of transitions made from state i to state j over the total number of transitions made out of state i. We have from problem 2 that γt(i) is the probability of being in state i at time t. Extending this, the probability of being in state i at time t and in state j at time t+1 can be calculated as follows:

𝜉! 𝑖, 𝑗 =𝛼! 𝑖 𝑎!"𝑏!𝛽!!! 𝑗

𝑃(𝑂|𝜆)

A good introduction to HMMs can be found in Rabiner's popular tutorial (Rabiner, 1989).

III. THE ALGORITHM We consider a set of users that consists of a mix of

positively biased, negatively biased, and fair users. We assume that time is slotted, and during each slot each user submits a rating about a service. Let T be the total number of slots. For simplicity, it is assumed that all the users submit a rating at each time slot. (This assumption can be easily removed, by appropriately modifying the way the αi(t) and βi(t) are calculated.)

For each user, we have a sequence of T ratings (observations) which we use to train an HMM (problem 3). The N hidden states of this HMM represent the different levels of quality under study, with 1 being the worst and N being the best. In addition the M ratings are defined so that 1 is the lowest and M is the highest. N and M are the same for all users.

Thus, we end up with as many HMMs as the number of users. Using statistical clustering techniques, we cluster the B matrices of these HMMs into two groups, one for fair and the other for biased users (both positively and negatively biased). This permits us to identify and filter out the unfair users. The HMMs belonging to the group of fair users are then aggregated into a single HMM, from which we can obtain the aggregate A matrix of the quality of the service. Also, using the HMMs of the fair users we can estimate the current level of the quality of the service.

We used the built-in MATLAB functions for HMM to implement the above algorithm and obtain numerical results.

To test the accuracy of the algorithm, we need user ratings from each of the three types of users. To achieve this, we define three λ's, all having the same A and π, but different B. The B matrix of the fair users should be such that the users' ratings echo the quality of the service. That is, if the service

has a very high quality, then the probability of the user giving a high rating should be high. Likewise, the user should give a low rating with a high probability if the quality of the service is low. For the other two types of users, however, the structure of the B matrix is different, since an unfair user tends to give the same (good or bad) rating, regardless of the actual quality of the service.

For each user, a sequence of T ratings is generated from its corresponding λ. Next, this sequence is used to estimate the A, B, and π matrices of the user. Using the B matrices, we cluster the users into two groups, i.e., fair and unfair users. For this, we take the mean squared error (MSE) of each estimated B matrix with the identity matrix, where

𝑀𝑆𝐸 =1𝑁

(!

!

𝑏!(𝑘) − 𝐼!(𝑘))!!

!!!

and bi(k) and Ii(k) are the (i,k)th element of the B and I matrices respectively.

The resulting MSE values are then classified into two clusters using the k-means clustering algorithm. As the identity matrix represents the truly fair matrix, the cluster with a center closer to 0 represents the group of fair users.

The A matrices of the filtered fair users can be then aggregated in order to estimate the A matrix of the quality of the service. Also, using the HMMs of the filtered fair users, we can calculate the most probable current quality level of the service.

IV. NUMERICAL RESULTS The results reported in this section were obtained assuming

that 60% of the users are fair, 20% are positively biased, and 20% are negatively biased. In addition, we assumed that the number of N=5, with 1 being the worst and 5 being the best, and the number of ratings M=5, with 1 being the lowest and 5 being the highest. (The selection of N=M=5 is not significant, and any other values can be readily used.)

We recall that A and π are the same for all users, and B varies according to the type of the user (negatively biased, positively biased, fair). In all experiments, we set the Markov chain to start with an equal probability of being in any of the 5 hidden states, i.e., π = [0.2, 0.2, 0.2, 0.2, 0.2]. Different A matrices were used in different experiments to reflect different scenarios of the quality of the service under study, namely, the quality oscillates around one of the 5 states, improving or worsening. Let Ai be the matrix A for the case where the quality oscillates around i. Then, we define the matrix as follows. Ai is a 5 by 5 matrix where the ith column is 0.9, that is, there is a 90% probability of being in the dominant state. All other entries are 0.025, making the sum of each row 1.

The ideal B matrix of a fair user, Bf, is the identity matrix I. However, to account for subjective deviations of the reviewers we used the following values:

𝐵! =

0.8 0.2 0      0 00.20

0.60.2

0.20.6

00.2

00

00

     0      0

0.20

0.60.2

0.20.8

For the positively biased and negatively biased B matrices,

Bp and Bn, we used the following values:

𝐵! =

0 0 0.1 0.1 0.800

00

0.10.1

0.10.1

0.80.8

00

00

0.10.1

0.10.1

0.80.8

𝐵! =

0.8 0.1 0.1 0 00.80.8

0.10.1

0.10.1

00

00

0.80.8

0.10.1

0.10.1

00

00

For each of the above Ai matrices, i = 1, 2, ..., 5, we define

three HMMs, namely, λf = (Ai, Bf, π), λp = (Ai, Bp, π) and λn = (Ai, Bn, π). Subsequently, we generate random samples of ratings for all the users, and then apply the algorithm described in the previous section to identify the group of fair users. Then, we compare the number of identified fair users to the original set of fair users and report the result as a percentage of correctly identified fair users.

The results are given in figures 1, 2 and 3. The three figures correspond to the three Ai matrices for i=1,2,3. In each figure, we give the percentage of correctly identified fair users along with the 95th confidence interval. For each figure, we assume 10, 50 and 100 users, and vary the number of ratings per user from 10 to 50. The confidence intervals are obtained by replicating each result 100 times, and each time using a different seed for the pseudo-random number generator. Because of the way the Ai matrices have been setup, the results for A4 and A5, are the same as those for A2 and A1, respectively, and hence they are not reported.

When the quality of service oscillates around state 3, the proposed method gives the best results, as shown in figure 3, ranging from 87% to 97% of correctly identified fair users. As we move towards the extreme cases of state 1 and 5, the accuracy decreases. This makes sense as it is not possible to distinguish the positive or negative (depending on the case) unfair users from the fair ones. An average of 65% of the fair users are identified in case of A1, and an average of 78% in case A2. These percentages maybe improved if we use additional information about the users, such as, membership to social groups, and/or additional information about the ratings, such as the two indices described in [1].

The results for the two cases of improving and worsening quality are not given as they are similar to the case where the quality is 5 and 1 respectively. This is probably due to the fact that the hidden Markov chain converges very quickly to state 5 for the improving case, and 1 for the worsening case.

Based on the results, we also observe that it is not necessary to have a large sample of users and ratings per user. We see

that with as little as 10 users and 10 ratings per user, we obtain results which are similar to those obtained with larger number of users and ratings per user.

Next we aggregate the A matrices of the fair users to a single 𝐴 matrix, by averaging out the one-step transition aij over all the A matrices, for each (i,j)th position. 𝐴  is the estimated Markov chain that governs the quality of the service.

Fig. 1. Prediction accuracy for A1

Fig. 2. Prediction accuracy for A2

Fig. 3. Prediction accuracy for A3

We tested the accuracy of 𝐴 by calculating the MSE of 𝐴 with each of the five original Ai matrices. The results are given in figures 4, 5 and 6. The confidence intervals were obtained

Fig. 4. MSE of 𝐴 with Ais when the matrix A of the system under study is A3. Number of users is 10.

Fig. 5. MSE of 𝐴 with Ais when the matrix A of the system under study is assumed to be A3. Number of users is 50.

Fig. 6. MSE of 𝐴 with Ais when the matrix A of the system under study is assumed to be A3. Number of users is 100.

as described above for the figures 1 to 3. As we can see, 𝐴  has the least error with the original Ai matrix used in the corresponding experiment.

Using the estimated HMMs of the filtered fair users, we can calculate  𝑠! ,  the most probable state of the quality of the service at time T. Specifically, for each filtered fair user u, u=1,2,…,U, where U is the total number of filtered fair users, we run the Viterbi algorithm using the user’s set of ratings and its estimated ΗΜΜ to obtain 𝑠!!, the most probable state at time T. Then, 𝑠! = (1/𝑈) 𝑠!!!

! .

Fig. 7. Average relative error for the most probable state at T.

Fig. 8. Average relative error for the most probable state at T, aggregated over number of ratings.

The accuracy of 𝑠! was tested as follows. For each user u we estimate 𝑠!! as above. Then, using the original HMM, i.e., (Ai, Bf, π), with the set of ratings of user u, we also calculated the most probable state at time T, 𝑠!!,. The difference between these two values is expressed by the relative error |𝑠!! −𝑠!!|/𝑠!!. We repeated this for all classified fair users and calculated the average relative error. The results are given in figure 7 for different number of ratings per user and for each matrix Ai, assuming 50 users. The confidence intervals were obtained in the same way as in figures 1 to 3. In figure 8, we plot the prediction accuracy aggregated over the number of ratings for different number of users (10, 50, and 100) and for each matrix Ai.

We notice interesting trends in figures 7 and 8. As we can see in figure 7, as the number of ratings increases per user, the prediction accuracy of the state at time T increases as well. The average prediction accuracy for 10 ratings is around 80% and for 50 ratings per user, it jumps to around 97%. However, there is no such similar improvement as we increase the number of users. As can be seen in figure 8, the average prediction accuracy for 10 users is around 91% and for 100 it is around 95%.

Finally, we turn towards a sensitivity analysis of our approach, i.e., how does the percentage of unfair users impact on the filtering accuracy. For this we varied the percent of unfair users from 0 to 50%. We assume that half of them are negatively biased and the other half positively biased. Keeping the number of users to 50 and the number of ratings per user to also 50, we calculate the average relative error (as in fig 7), for (Ai, Bf, π), i=1,2,3. The results are presented in Table 1. As the number of negatively and positively biased users is the same, the results of A4 and A5 mimic the behavior of the results for A2 and A1 respectively and hence are omitted from the table.

TABLE 1

AVERAGE RELATIVE ERROR

Percentage of unfair users 0% 10% 20% 30% 40% 50%

A1 1 0.96 0.94 0.91 0.9 0.87 A2 1 0.97 0.96 0.93 0.91 0.89 A3 1 0.98 0.97 0.96 0.93 0.9

We make the following observations. First, as the number

of unfair users increases, the relative error increases as well. This not only makes intuitive sense but it can also be explained using our HMM approach. The number of filtered fair users is directly proportional to the number of original fair users. Since we are using the filtered fair users to estimate the original HMM, the lesser the number of filtered fair users, the less accurate will be the estimated HMM. Resultantly, the calculation for  𝑠! ,  the most probable state of the quality of the service at time T, will also be less accurate. Second, the HMM with A3 is the least affected as the number of unfair users increase. This due to the fact that A3 results in an inherently better cluster of fair users (as explained in the algorithm), and it is the least affected as the number of unfair users increases. Overall the filtering algorithm still works satisfactorily.

V. CONCLUSION In this paper, we propose a method based on Hidden

Markov Models (HMMs) for filtering out users who submit unfair ratings to an online reputation system. We show through experimentation that this filtering method is quite accurate, particularly when the quality of the service oscillates around a point that is away from the two extreme cases of very bad and excellent quality. Based on the estimated HMMs of the filtered fair users, we calculate 𝑨, the estimated Markov chain that governs the quality of the service, and the current state of quality of the system. We show that these two results have a good accuracy. Finally, we demonstrated that our

filtering algorithm has a good accuracy when we vary the number of unfair users from 0% to 50%.

REFERENCES [1] W. Fan and H. Perros, "A reliability-based trust

management mechanism for cloud services.," Trust, Security and Privacy in Computing and Communications(TrustCom), 12th IEEE International Conference, pp. 1581-1586, July 2013.

[2] C. Dellarocas, "Immunizing online reputation reporting systems against unfair ratings and discriminatory behavior.," 2nd ACM Conference on Electronic Commerce, pp. 1581-1586, October 2000.

[3] Y. Yang, Y. Sun, S. Kay and Q. Yang, "Defending online reputation systems against collaborative unfair raters through signal modeling and trust.," Proceedings of the 2009 ACM symposium on Applied Computing, pp. 1308-1315, March 2009.

[4] J. Zhang and R. Cohen, "Trusting advice from other buyers in e-marketplaces: the problem of unfair ratings.," Proceedings of the 8th international conference on Electronic commerce, pp. 225-234, 2006 August.

[5] Y. Liu, Y. Yang and Y. L. Sun, "Detection of collusion behaviors in online reputation systems.," Proceedings of 42nd IEEE Asilomar Conference on Signals, Systems and Computers, pp. 1368-1372, October 2008.

[6] A. Josang and R. Ismail, "The beta reputation system.," 15th Bled Electronic Commerce Conference, June 2002.

[7] M. Moe, M. Tavakolifard and S. Knapskog, "Learning trust in dynamic multiagent environmnets using hmms.," Proceedings of the 13th Nordic Workshop on Secure IT Systems (NordSec 2008), January 2008.

[8] L. Chang and Z. Jiliu, "The reputation evaluation based on optimized hidden markov model in e-commerce.," Mathematical Problems in Engineering, volume 2013. Hindawi, 2013.

[9] E. ElSalamouny, V. E. Sassone and M. Nielsen, "Hmm-based trust model, in: Formal aspects in security and trust.," Lecture Notes in Computer Science Volume 5983, pp. 21-35, 2010.

[10] H. Yahyaoui and S. Zhioua, "Bootstrapping trust of web services based on trust patterns and hidden markov models.," Springer, September 2012.

[11] L. R. Rabiner, "A tutorial on hidden markov models and selected applications in speech recognition.," Proceedings of the IEEE, VOL.77, NO. 2, February 1989.