A Random Walks Method for Text Classification

YUNPENG XU1, XING YI2, CHANGSHUI ZHANG1

1State Key Laboratory of Intelligent Technologies and Systems,

Tsinghua University, 100084 Beijing, P.R. China2Center for Intelligent Information Retrieval, Department of Computer Science,

University of Massachusetts, Amherst, MA01003, USA

E-MAIL:xuyp03@mails.tsinghua.edu.cn, zcs@mail.tsinghua.edu.cn, yixing@cs.umass.edu

Abstract

Practical text classification system should be able to uti-

lize information from both expensive labelled documents and

large volumes of cheap unlabelled documents. It should also

easily deal with newly input samples. In this paper, we pro-

pose a random walks method for text classification, in which

the classification problem is formulated as solving the ab-

sorption probabilities of Markov random walks on a weighted

graph. Then the Laplacian operator for asymmetric graphs

is derived and utilized for asymmetric transition matrix. We

also develop an induction algorithm for the newly input doc-

uments based on the random walks method. Meanwhile, to

make full use of text information, a difference measure for

text data based on language model and KL-divergence is pro-

posed, as well as a new smoothing technique for it. Finally

an algorithm for elimination of ambiguous states is proposed

to address the problem of noisy data. Experiments on two

well-known data sets: WebKB and 20Newsgroup demon-

strate the effectivity of the proposed random walks method.

1 Introduction

Text classification, an important research area in datamining and knowledge discovery, has broad applica-tions such as web information mining, junk mails filter-ing, personal news recommendation system, documentdatabases organization, etc. [10] [11] are referred to astwo comprehensive surveys on text classification prob-lems. Considering the fact that labelling documentsis both costly and time-consuming, we will follow theSemi-Supervised Learning approach where there is asmall number of labelled documents and large volumesof unlabelled documents. Over the past few years, manysemi-supervised algorithms have been proposed with ap-plication to different problems. A comprehensive surveyup to 2001 can be found in [12], or a more recent one in[20].

In a practical text classification system, manythings need to be carefully considered such as how todesign efficient classification algorithms, how to extract

as much information from the text as possible, how toefficiently deal with new coming documents, etc. In thispaper, we propose a random walks method for text clas-sification, which consists of two aspects: random walksmodel and graph representation for text. Random walksmodel formulates the classification as a problem of solv-ing the absorption probability of random walks on aweighted graph. More specifically, labeled documentsare viewed as the absorbing states while unlabeled doc-uments as the non-absorbing states. Then the predictedlabels of unlabeled documents are determined by theprobabilities that they are absorbed by different absorb-ing states. In this method, the Laplacian operator forasymmetric graphs is derived so that asymmetric transi-tion matrix can be used for learning. Then an inductionalgorithm is developed for dealing with new documentsbased on this method. In the usual graph representationof text data, each document is represented as a vertexon the graph, in which the most important informationis the distance between two vertices. Then in this pa-per a new distance measure for the text data based onlanguage model [8] [13] and KL-divergence is definedin order to better capture the difference between docu-ments. In the end we discuss the smoothing techniqueand also the elimination algorithm of ambiguous statesto address the problem of learning with noisy data.

The paper is organized as follows. Section 2 intro-duces the random walks model. Section 3 describes thegraph representation for text. Experimental results areshown in Section 4. Section 5 concludes and discussesour future work.

2 Random Walks Model

Firstly, some terminologies and notations are given. Agraph is a pair G =< V,E > which consists of verticesv ∈ V and edges e ∈ E ⊆ V ×V . A finite states graphis a graph with finite vertices. A graph is weighted,if each edge is associated with a value wij ∈ R, whichdoes not necessarily satisfy wij = wji. The degree di of

338

vertex vi is di =∑

j wij , i.e., the summation of weightsof all edges that are connected to vi. The LaplacianMatrix of the graph is defined to be L = D −W [7],where W is the weight matrix, D is a diagonal matrix,whose ith diagonal element is di.

2.1 Random Walks for Semi-supervised learn-ingRandom walks is a stochastic model on a weighted fi-nite states graph, which exploits the structure of datain a probabilistic way. Here, we consider its applica-tion in semi-supervised classification problems. Givensuch a graph, suppose there are l labeled vertices V L ={v1, v2, ..., vl} with labels fL, and u unlabeled verticesV U = {vl+1, vl+2, ..., vl+u}, the task is to predict thelabels fU of V U . fL and fU are from the same labelset G = {g1, g2, ..., gK}. The random walks approach tothis problem is direct and natural: it regards all the ver-tices on the graph as different states of a Markov chain.The one-step transition probability pij from vi to vj isdefined by

(2.1) pij =wij

di,

or written in the matrix form

(2.2) P = D−1W.

The label fi of the vertex vi ∈ V U is then determinedby the probabilities that it reaches different labeledvertices, i.e., if the summation of the probabilities thatit reaches the vertices with label gk is the largest, it isassigned label gk.

In this paper, we consider a Markov chain withabsorbing states, which assigns different roles for la-beled and unlabeled data respectively. More specifically,the labeled samples are viewed as the absorbing stateswhile the unlabeled samples as the non-absorbing states,which means the former will only transit to themselveswith probability 1, whereas the latter are free to transitto any other states. Therefore once the unlabeled sam-ples reach the absorbing states, they will be absorbedand will not move on. Then the label fi of the vertexvi ∈ V U is determined by the probabilities that it isabsorbed by different labeled vertices in equilibrium.

Let P be the one-step transition matrix of thechain. For convenience, we partition the matrixaccording to the label status of the vertices as

(2.3) P =(

I OPUL PUU

),

where I is an l× l identical matrix, O is an l×u matrixwith all 0 elements, PUL is the transition probability

that non-absorbing states transit to absorbing states,PUU is the transition probability for non-absorbingstates. It is known [6] that the absorbing probabilitiesthat non-absorbing states are absorbed by absorbingstates in equilibrium is given by

(2.4)Pr(V L|V U ) = (I − P−1

UU )PUL

= L−1UUWUL

where LUU and WUL are corresponding blocks of Land W . Therefore, the probabilities of the unlabeledsamples being absorbed by the labeled sample set Ak

with label gk is

(2.5) Pr(Ak|V U ) = L−1UUWULfk,

where fk is an l × 1 vector which satisfies fk,i ={1 fL

i = k0 fL

i 6= k. Then the labels of the unlabeled samples

are given by

(2.6) fU = arg maxk{Pr(Ak|V U )}.

2.2 Review of Related WorkThe idea of applying random walks model to semi-supervised learning is not new. In [14], Szummeradopted a Markov random walks model withoutabsorbing states. The model assumes that for eachdata there exists a label distribution, the parameters ofwhich are then estimated by EM algorithm. However,there are three major differences between [14] and themodel used here. First, [14] has no absorbing states.Second, [14] considers the situation after a number oftransitions, therefore crucially depends on the choiceof the time parameter. Instead we consider the chainin an equilibrium state. Third, the values of the labelson the labeled data in [14] are not fixed, but herethe labeled data are regarded as absorbing states andtherefore have fixed values. [19] proposed a regulariza-tion framework based on Gaussian field and Harmonicfunctions, which is closer to our approach. They gotthe same label function as formula 2.5 but with dif-ferent interpretation, which is not surprising since theestimation of data labels using random walks can alsobe regarded as a Dirichlet Problem to find a harmonicfunctions having the boundary values[6]. However, ourrandom walks interpretation has a better formulationof the problem, and is easier to be generalized tomulti-class problems and out-of-sample problems. In[6], Doyle also introduced the interesting connectionbetween random walks and electric networks.

339

2.3 Laplacian Operator for AsymmetricGraphsIn a real problem, the transition matrix P of ran-dom walks does not necessarily satisfy symmetry, i.e.,pij = pji might be violated. Correspondingly, the graphweights do not necessarily satisfy wij = wji. Graphswith asymmetric weights are termed AsymmetricGraphs. However, for learning problem asymmetry informula 2.5 is a trouble, since LUU defined in symmetricway might not be semi-definite, which would probablyhave negative eigenvalues and therefore produce nega-tive absorbing probabilities. To address this problem,we will show in the following that asymmetric graphsstill have Laplacian matrices in symmetric form. Ourproof is given in the framework of graph analysis[18].

Theorem 2.1. The Laplacian matrix for asymmetricgraph is given by L = D −M , where Mij =

√WijWji.

Proof. We start the proof from the definitions of somebasic operators on a discrete graph. Consider twoarbitrary vertices vi, vj ∈ V , and the edge eij ∈ Ebetween them. Let H(V ) be the Hilbert space of real-value function endowed with the usual inner product< ϕ, φ >=

∑vϕ(v)φ(v), where ϕ and φ are two arbitrary

functions in H(V ). Similarly, we define H(E) and let ψbe an arbitrary function in it.

In graph analysis, the graph gradient is defined tobe

(2.7) ∇ϕ(vi, vj) =√wijϕ(vi)−

√wjiϕ(vj)

for both symmetric and asymmetric graphs. It is clearthat ∇ϕ(vi, vj) = −∇ϕ(vj , vi) always holds. Thegraph divergence (divψ)(vi) measures the net outflowof function at vertex vi and is defined by the followingadjoint operator

(2.8) < ∇ϕ,ψ >=< ϕ, divψ > .

Plugging the expression for graph gradient into formula2.8 and expand the latter, we have

(2.9) (divψ)(vi) =∑

j

√wij(ψij − ψji),

where the summation is over all vj that are connectedto vi. By virtue of the definition of Laplacian operator

(2.10) ∆ϕ = div(∇ϕ)

and substitute the expressions for graph divergence andgraph gradient, it follows that

(2.11) (∆ϕ)(vi) = (diϕ(vi)−∑

j

√wijwjiϕ(vj)).

Directly from formula 2.11, we obtain the Laplacianmatrix for asymmetric graph

(2.12) L = D −M,

where Mij =√WijWji. �

Formula 2.12 demonstrates that the Laplacian ma-trix of an asymmetric graph still has an explicit defini-tion. On the one hand, M is the geometrical averageof the weight matrix W and its transposition. There-fore, it is symmetric and has no computational problemin solving formula 2.5. On the other hand, the degreematrix D is computed from the asymmetric W insteadof M , therefore still possesses the original asymmetricinformation.

2.4 Random Walks InductionIn practise, classifiers are not expected to be retrainedeach time when new documents are input. Instead, theyshould be able to easily deal with the new input data.This capacity of the classifier is said to be InductiveLearning or Induction.

Some works have investigated the problem of Induc-tion. In [4], Chapelle proposed to approximate a newtest data with observed data, which is not quite direct.In [16], Yu approximated the label function with the lin-ear combination of a set of basis functions which is ableto handle new test points. But basis functions shouldbe carefully chosen to guarantee a good result. In [1],expansion to new points depends on the RepresenterTheorem of RKHS. This approach is complicated andgood cost functions and parameters should be carefullyselected. Here we propose an induction algorithm basedon the Markov random walks method, which is natural,simple and effective.

The algorithm is based on a simple assumption: thenewly input data will not distort the transition relationsamong the observed data. In fact, it also holds implicitlyin[1] [4] [16]. This means that the observed data willnot transit to these new data. Therefore, we have thefollowing augmented transition matrix

(2.13) P =

I O oPUL PUU oPNL PNU o

,

where PNL and PNU are the transition probabilitiesthat the new data transit to the labeled data V L andunlabeled data V U , respectively. For a new data vN

and an observed data vi ∈ V L ∪ V U , the transitionprobability from vN to vi is determined by their edgeweight, i.e.,

(2.14) pNi =wNi∑

j∈V U∪V L

wNj.

340

Then we have the probability that vN is absorbed bythe absorbing states Ak with label gk

(2.15) pr(Ak|vN ) =∑

j∈V U

pNjpr(Ak|vj) +∑

j∈Ak

pNj ,

where pr(Ak|vj) is the probability that vj ∈ V U isabsorbed by Ak. We can further write formula 2.15in the following matrix form

(2.16) pr(Ak|vN ) = (PNUL−1UUWUL + PNL)fk.

where, L−1UU can be computed and saved by the trans-

ductive learning introduced in section 2.1, therefore noextra inversion is needed. Similarly, the labels of thenew data are given as

(2.17) fU = arg maxk{Pr(Ak|vN )}.

Our induction algorithm is designed based on theintuitive interpretation of the random walks modelin the framework of random walks. The algorithmutilizes both local information via one-step transitionto labeled points and global information via transitionto unlabeled points. Consequently, it is not a simpleapproximation of nearest neighborhoods. In termsof computational complexity, this induction algorithmrequires no more inversion of matrix. The only matrixoperations we need to implement are the multiplicationand addition. Therefore, the algorithm is fast and hassignificant advantage on computation.

In figure 1, we give a toy example on two moondata to test the random walks model and the inductionalgorithm. The observed data points are plotted outusing small dots. For each class, we randomly labelone data, which is marked with small triangle. Theclassification results are described by using differentcolors. Classification boundary is determined using theabove induction algorithm on data that are randomlysampled from the data space. It can be seen that theproposed algorithm can successfully classify both theobserved data and the newly input data.

3 Graph Representation for Text

When adopting the random walks model, data shouldbe represented as vertices on the graph. The most im-portant information concerning this graph representa-tion is the distance between data. Therefore, in orderto guarantee a satisfying classification result, we needfirst precisely measure the distances between differentdocuments. In this section, we propose the Text Diver-gence for measuring text difference based on languagemodel and KL-Divergence. In experiments, a relevantUniform Laplace Smoothing technique is utilized andan Ambiguous State Elimination Algorithm is proposedto remove noisy data.

Figure 1: A toy example for the random walks modeland its induction algorithm.

3.1 Text DivergenceLet Ti and Tj be two arbitrary documents. Whenmeasuring the difference between these two documents,we do not just regard them as two different sets ofwords. Instead, we think their words are generated bytwo different word distributions. In fact, the statisticalLanguage Model, which is a probabilistic mechanismfor generating text, views a document in the sameway, i.e., document is generated by a certain ”source”,which corresponds to the word distribution here. Itis natural that we would adopt the KL-Divergence asa measure of difference between the two documentssince it can capture well the difference of distributionsbetween these two documents. Therefore we define theText Divergence (TD) to be

(3.18)TD(Ti, Tj) = KL(Ti, Tj)

= H(Ti||Tj)−H(Ti||Ti)

where H(Ti||Tj) is the cross entropy between two dis-tributions, i.e.,

(3.19) H(Ti||Tj) = −∑w

Pr(w|Ti) log(Pr(w|Tj)),

where summation is over all words in the vocabulary,including both occurred ones and un-occurred ones. Wewill discuss the smoothing technique for the un-occurredwords in the next section.

Strictly speaking, TD is not a distance measure,since it is not necessarily symmetric. This is why weuse difference as above. However, we can still use itgiven the method introduced in section 2.3.

Once TD is computed, it follows the weight between

341

two documents on the graph, which is

(3.20) wij = exp{−TD(Ti, Tj)2

2σ2},

where σ is estimated using an exhaustive search on thetraining set in our approach. Of course, more sophis-ticated learning methods can be adopted if necessary.Furthermore, weight matrix W should better be sparseso as to depress noise and to alleviate computationalburden. We therefore regulate that wij is non-zero onlyfor Tj ∈ Nb{Ti}, where Nb{Ti} denotes the set of near-est neighbors of Ti.

3.2 Uniform Laplace SmoothingTo compute TD between two documents, we need toestimate the probability of each word in the documents,which is defined to be

(3.21) Pr(wk|Ti) =#(wk)Ti

#(w)Ti

,

where #(wk)Tiis the total number of times that word

wk occurs in Ti, #(w)Tiis the total number of times

that all words occur in Ti. However, the so-called ”Zero-Frequency Problem” will happen if some words do notoccur in the document. Therefore, smoothing tech-niques are needed. The most frequently adopted tech-niques include Laplace Smoothing, Lindstone Smooth-ing, Good-Turing Estimate and some interpolationmethods. We refer [5] as a comprehensive empiricalstudy on smoothing.

Here, we consider the Laplace Smoothing, whichadds a small constant ε to each count and then normal-ize. ε is usually 1 or smaller, and has the same value forall documents. Then the word probability is given by

(3.22) Pr(wk|Ti) =#(wk)Ti

+ ε

#(w)Ti+ #word× ε

,

where #word is the number of different words, i.e., thedimension of text features. However, this simple tech-nique has a rather poor performance in the applicationhere because the additive constant would probably dis-tort the original word distributions. What is more, thedistortion would dramatically vary over all documents,which severely affects the estimation of TD.

We expect the adopted smoothing technique cansuccessfully solve the zero frequency problem withoutmuch disturbance to the word distribution. There-fore, we propose the following Uniform Laplace Smooth-ing(ULS). The key idea is very simple: each un-occurredword in all documents should be smoothed to the samesmall probability δ. δ should be small enough such thatthe disturbance to word distribution is negligible. Here

we set it to be one-tenth of the smallest occurred word’sprobability of all documents. The laplace additive con-stant ε is then determined by δ, which is

(3.23) ε =#(w)Ti

× δ

1−#word× δ.

For a given document, ε is fixed. But different docu-ments could have different ε, i.e., documents with largerword size could have larger ε, and vise versa. Thissmoothing technique can properly allow for the varianceof different documents.

3.3 Ambiguous States EliminationData from a real problem are usually noisy. Some sam-ples may have uncertain labels or even multiple labels.For example, in the news data, it is not always easyto differentiate documents from the politics topic andthose from the economic topic. Some news may con-tain information concerning both topics since the twoare frequently related. If we want to classify news fromthese two topics, such ambiguous samples in the train-ing set would probably lead to a poor classification re-sult. We illustrate the problem with figure 2 a. Eachdata point in the graph selects its seven nearest neigh-bors as its next possible transition states. However,the noise data, which is enclosed by red squares, can-not find their neighbors correctly. Such data are termedAmbiguous States. These states are troublesome in ran-dom walks: they might mistakenly transit to the sam-ples with wrong labels and consequently cause errors inclassification.

To improve the accuracy of random walks classifi-cation, we need to eliminate the ambiguous states. Theambiguity of a sample is determined by its local infor-mation. Intuitively, samples with low ambiguity should”fit” well with their neighbors. Here, ”fit” means thatsamples should be close to their neighbors and their lo-cal shapes should be similar. By contrast, ambiguousdata are sparse. They have larger distances to theirneighbors and have distinct local shapes, therefore dif-ferent local information. The local information used inour method is the eigenvalue of local covariance matri-ces. It is clear that ambiguous states should have largerlocal eigenvalues than their neighbors’. Therefore, sim-ilar to the inadaptability defined in [17], we define theambiguity of sample vi to be

(3.24) Ambi =∑

j

λij

λij

,

where λij is the jth eigenvalue of the local covariancematrix of vi, λij is the average of λkj over all vi’sneighbors vk ∈ Nb{vi}.

342

a b

Figure 2: Illustration of ambiguous states and their elimination.

The computation of local covariance matrix needsthe original feature representation. However, in someoccasions we only have the graph representation ofdata. Fortunately, it’s not difficult to solve the localeigenvalues without local covariance matrix. In fact,MDS is the method to find the embedding coordinatesfrom the distance matrix [3]. Therefore, when onlythe local distance matrix H of vi is available, the localeigenvalues can be obtained through decomposition ofthe matrix −NH̃N , where H̃ is the matrix for squareddistance, N = I − 1 × 1T /n, 1 is a vector of n ones, nis the number of neighbors.

In figure 3, we give the ambiguities of all samplesin the course and faculty subsets of the WebKB data.Then the ambiguous states can be eliminated by settinga proper threshold, or the number of samples to beeliminated. For example, in Figure 3, we get a thresholdshown by the green line if 2% samples are to be removed.The elimination of the ambiguous states would clearlyimprove the neighborhood graph construction, thereforereduce the possibility of false transition among samples,as shown in figure 2 b.

4 Experiments

In this section, we validate the proposed framework withexperiments on two well known real data sets: WebKBdata and 20Newsgroup data.

4.1 WebKB Data1

In WebKB data, there are overall 8,282 web pagesfrom 4 colleges, which have been manually classified into

1This data is available at http://www.cs.cmu.edu/afs/cs.cmu.

edu/project/theo-20/www/data/webkb-data.gtar.gz

Figure 3: The ambiguities of all samples in the courseand faculty subsets of the WebKB data, where thelargest six eigenvalues are used in formula 3.24. Thehorizontal line is the threshold if 2% samples are to beeliminated.

343

a b

Figure 4: Experimental results on WebKB and 20Newsgroup data. ”RW Transd + KL” denotes the result givenby random walks transductive learning using the text divergence measure, ”RW Induct + KL” denotes the resultgiven by random walks inductive learning using the text divergence measure, ”RW Transd + dot” denotes theresult given by random walks transductive learning using the dot distance measure, ”RW Induct + dot” denotesthe result given by random walks inductive learning using the dot distance measure.

7 categories. Here, we consider its two subsets: courseand faculty, which have 2054 pages in total. To extractand select document features, we implement the follow-ing operations. First we remove 500 common wordsfrom the standard stop-list. We then adopt the ”Doc-ument Frequency + χ2 statistics Max Cut” [9] method,which results in 5000 demensional document featurevectors. We randomly divide the dataset into three sub-sets: labeled set, unlabeled set and ”unseen” set (totest the induction algorithm). The unlabeled set hasa fixed size of 500 documents. In experiment, we varythe size of labeled set and put the left documents intothe ”unseen” set. Each document chooses 15 nearestneighbors as its next possible transition states. Wheneliminating ambiguous states, 2% unlabeled documentsare removed.

When different number of documents are labeled,we obtain the following classification results, as shownin figure 4 a, where each point is the average over 30random runs. The red line corresponds to the resultsgiven by random walks transductive learning on the un-labeled data, while the blue line corresponds to the re-sults given by random walks inductive learning on the”unseen” data. It shows that the transductive learn-ing performs a little better than the inductive learning,but the difference is not obvious. For comparison, weconsider SVM, which is known for its superior perfor-mance on high dimensional and small labeled problems.Here, RBF kernel is adopted with its parameter beingestimated using an exhaustive search on the training set.

The classification results are shown using the green line.It can be seen that SVM has a better performance whenthere are too few labeled samples. But as the number oflabeled data increases, random walks will surpass SVMvery soon. Besides, we also provide classification re-sults of random walks transductive learning and induc-tive learning using the common symmetric dot distancemeasure, which are shown using the magenta and cyanlines respectively. This proves the superb effectivenessof our text divergence measure and its smoothing tech-nique.

4.2 20Newsgroup Data2

In 20Newsgroup data, there are overall 20,000 newspieces, which have been manually classified into 20 cat-egories, i.e., around 1000 documents per class. Here,we consider its 4 subsets: autos, motorcycles, base-ball, hockey, which have 4000 documents in total.We adopted the same feature extraction and selectionmethod as the above experiment, as well as the sameexperimental settings. After 30 independent runs, weget the above experimental results, as shown in figure4 b. Clearly, similar conclusions can be drawn exceptthat SVM never exceeds random walks even when thenumber of labeled data is small.

2This data is available at http://people.csail.mit.edu/ jren-

nie/20Newsgroups/

344

5 Conclusions and Future Work

We propose in this paper a random walk method fortext classification. In this method, classification is for-mulated to the problem of solving the probabilities thatunlabeled documents are absorbed by labeled docu-ments. Based on it, we derive the Laplacian operator forasymmetric graphs to solve the computational problemcaused by asymmetric transition matrix. We then de-veloped a random walks induction algorithm to expandthe method to newly input documents. On the otherhand, for better representing text data, the method pro-poses a new text difference measure based on statisti-cal language model and KL-Divergence, and a relevantsmoothing technique. Compared with the common dotdistance measure, such a measure can better mine thedifference information from the text data. We also pro-pose an ambiguous states elimination algorithm to ad-dress the problem of noisy input data. Experiments onWebKB and 20Newsgroup data sets show that, theproposed method has a superb performance, which ishigher than those of SVM and the random walks usingdot distance measure.

However, there are still some problems with theproposed method. First, although the algorithm forambiguous state elimination can eliminate noisy data,it might also remove some non-noisy data. Our nextwork is to make this algorithm more robust. Second,we need more experiments to compare the proposedmethod with other semi-supervised methods, such asTransductive SVM[15], and MinCut[2]. Third, it isinteresting to consider active learning based on randomwalks, which we will investigate later.

Acknowledgements

This work is supported by the Project (60475001) ofthe National Nature Science Foundation of China. Wewould like to thank Yungang Zhang, Shijun Wang, JingYang, Mikhail Belkin, and Dengyong Zhou for their helpwith this work. Special thanks to Kai Yu, who providedus his data, which is very helpful to our experiments.We would also thank the anonymous reviewers for theirvaluable comments.

References

[1] M. Belkin, P. Niyogi, V. Sindhwani, On ManifoldRegularization, AI & Statistics, 2005.

[2] A. Blum, S. Chawla, Learning from Labeled and Un-labeled Data using Graph Mincuts, Proc. 18th Interna-tional Conf. on Machine Learning, 2001.

[3] I Borg, PJF Groenen, Modern multidimensional scal-ing: theory and applications, Springer, New York, 1997.

[4] O. Chapelle, J. Weston, B. Scholkopf, Cluster kernelsfor semi-supervised learning, Advances in Neural Infor-mation Processing Systems 15, 2003.

[5] S. F. Chen, J.T. Goodman, An empirical study ofsmoothing techniques for language model, In Proceed-ings of the 34th Annual Meeting of the ACL, 1996.

[6] P.G. Doyle, J. Snell Random Walks and Electric Net-works, Mathematical Assoc. of America, 1984.

[7] R. Merris, A survey of graph Laplacians, In Lin.Multilin. Algebra, 39 (1995), 19C31.

[8] J.M. Ponte, W.B. Croft, A language modeling approachto information retieval, Proceedings of ACM-SIGIR1998, pages 275-281, 1998.

[9] M. Rogati, Y. Yang, High-performing feature selectionfor text classification, In Proceedings of the eleventhinternational conference on Information and knowledgemanagement, McLean, Virginia, USA (2002)

[10] F. Sebastiani, Machine Learning in Automated TextCategorization, ACM. Computing Surveys, Vol. 34, No.1, March 2002, pp.1-47.

[11] F. Sebastiani, Text Categorization, In Text Mining andits Applications to Intelligence, CRM and KnowledgeManagement, pages 109-129, WIT Press, Southamp-ton, UK, 2005.

[12] M. Seeger, Learning from Labeled and Unlabeled Data,Technical Report, University of Edinburgh, 2001.

[13] F. Song, W.B. Croft, A general language model for in-formation retrieval, In Proceedings of Eight Interna-tional Conference on Information and Knowledge Man-agement. (CIKM1999), 1999.

[14] M. Szummer, T. Jaakkola, Partially labeled classifica-tion with Markov random walks, Advances in NeuralInformation Processing Systems, 14, 2002.

[15] V. N. Vapnik, Statistical Learning Theory, J. Wiley,New York, 1998.

[16] K. Yu, V. Tresp, D. Zhou, Semi-supervised Induction,Advances in Neural Information Processing Systems17, 2005.

[17] Y. Zhang, C. Zhang, S. Wang, Clustering in KnowledgeEmbedded Space, In ECML03, 2003.

[18] D. Zhou, B. Scholkopf, Transductive Inference withGraphs, Technical report, (2004).

[19] X. Zhu, Z. Ghahramani, J. Lafferty, Semi-supervisedLearning using Gaussian Fields and Harmonic Func-tions, In ICML-2003, 2003.

[20] X. Zhu, Semi-Supervised Learning Literature Survey,Technical Report 1530, Computer Sciences, Universityof Wisconsin-Madison, 2005.

345