26
, Peking University, Beijing, China Technical Report PKU-CS-NCIS-TR2011XX June 2011 Computer Networks and Distributed Systems Lab Peking University Beijing, China 100871

ÌÌÌKKK˝˝˝333'''˛˛˛ ÷÷÷¥¥¥˙˙˙AAA^^^net.pku.edu.cn/~zhaoxin/Topic-model-xin-zhao-wayne.pdfComputer Networks and Distributed Systems Lab Peking University Beijing, China

  • Upload
    others

  • View
    0

  • Download
    0

Embed Size (px)

Citation preview

Page 1: ÌÌÌKKK˝˝˝333'''˛˛˛ ÷÷÷¥¥¥˙˙˙AAA^^^net.pku.edu.cn/~zhaoxin/Topic-model-xin-zhao-wayne.pdfComputer Networks and Distributed Systems Lab Peking University Beijing, China

ÌÌÌKKK���...333©©©������÷÷÷¥¥¥���AAA^

ëëëccc,ooo¡¡¡²²²

Peking University, Beijing, China

Technical Report PKU-CS-NCIS-TR2011XXJune 2011Computer Networks and Distributed Systems LabPeking UniversityBeijing, China 100871

Page 2: ÌÌÌKKK˝˝˝333'''˛˛˛ ÷÷÷¥¥¥˙˙˙AAA^^^net.pku.edu.cn/~zhaoxin/Topic-model-xin-zhao-wayne.pdfComputer Networks and Distributed Systems Lab Peking University Beijing, China

ÌÌÌKKK���...333©©©������÷÷÷¥¥¥���AAA^

ëëëccc,ooo¡¡¡²²²

Computer Networks and Distributed Systems Lab, Peking UniversityEmail: [email protected], [email protected]

Abstract

ÌK�.´?c5©��÷¥Ñy��«VÇ�.§Ø2�DÚ��m�þ�.Ú�ó�.@�§�üX/�Ä©�3c;�mþ��ݧ ´Ú\ÌK�m§l ¢y©�3ÌK�mþ�L«§z��ÌK´��3c;�mþ�VÇ©Ù"ÏLÚ\ÌKù�Vg§�±�5ü�Ð?µ1¤¢y©��$�L«¶2¤Ä�©�8ÜþÛ¹�Â��÷§=ÌK"glDavid BleiJÑLatentDirichlet Allocation£LDA¤�§TØ©®²�Ú^2836õg1§ÌK�.�A^A�CX©��÷Ú&E?n�¤k+�"ÌK�.���êÆÄ:Ü©��E,§�nã¦þ£;� H qE,�úªí�§l©��÷��Ý5©ÛÌK�.§�?�Ún)ÚA^ÌK�.���Á=ó�"

'''���ccc ÌK�.§©��÷§&E?n

1 ÚÚÚóóó

ÑyÌK�.c§&E?nÚ©��÷+�3©�L«þÌ��Ê331¤�m�þ�. [78]¶2¤ÚO�ó�. [71]"�m�þ�.{ü!´Ã¿�3¢SA^¥�~k�§glJÑ�§��4��A^§8c��¤Ù�û�|¢Ú�£Xgoogle!zÝ�¤Úm ���5��ó�.óä£XLucene2�¤E,æ^Ù��u¢�.¥�©�L«�{"ÚO�ó�.´Cc5�XÚOVÇ�×�uÐ Ñy�§§kX�~j¢�êÆÄ:��|±§¿��~N´*Ð!K\�õ�&EA�"¦+üö3�{ØþkXØÓ§��´Äu�5�ê�AÛCz§��´ÄuÚO�{�VÇ©Ù§üöPkXéõ�Ó:§Ù¥��§�´�­��Ò´üö@�©�Ñ´3c;�mþ?1L«�§�Ò´`��©�¬/¤���éõ£©�→c¤�N�½öL«"�X<�éu©�@£�uЧ<�m©J¦Ø©������n)§l ¦�O�Å$�<�

U�Ð/“n)”©�"��¡§Ñy�'��\�©��÷½ög,�ó?Ö§XgÄ<ó¯�"Ó�<��m©J¦�Lk“Lyå”�©�L��ª£XÛ�÷©�d3�“�”¤"Ù¥d3�©Û(Latent Semantic Analysis) [43]Ò´��@Ï��Ló�§�5�ÌK�.¢Sþ�E,ò^ÙØ%g�"d3�©ۧ�»± <�éu©�L«���g�½ªµ©�´L«3c;�mþ�"d3�©ÛKM#ª/Ú\�Â�ݧ�Â�Ý´©�8Üþ&E�ß L«§ ©�K´3ù�Â�Ýþ���L«"/�/`§3± �©�→cN�L«¥§Ú\���Â�ݧ=©�→�Â→c"d3�Â���{´�Äc�c3©�¥��y§,�ÏL�5�ê��{5J�Ñù“�”�ݧ,�¢y©�3�Â�mþ�$�L«"$�L«½ö`ü�´DÚêâ©Û¥���­�¯K [5]§Ù8�Ò´��é���éuêâ�äkLyå!�Ø �L«�{§$�L«k�Ð?§XØ�DÑ�K�!ü$êâL«¤���"�XVÇÚO©Û3©�ï�A^�ØäuЧd3�©Ûl�5�ê�©Û�ª�?�ÚJ,

�VÇÚO�©Û�ª [37](pLSI½öpLSA) 3"�cz��Â�ÝéA��A��þ§3VÇ�.¥§z��Â�ÝtKéA���c;þV�VÇ©Ù§={Pr(w|t)}w∈V¶©�éuz��Â�Ý��­§éA�VÇ�.¥§òz�©�dL«¤���Â�mþT�VÇ©Ù§={Pr(t|d)}t∈T"@Ï�VÇÌK

�.�¡�“aspect model”"òd3�©Û�VÇÿÐ��±�5A�Ð?µ1¤�±N´Ú\�õ�&E§Xk�&E¶2¤��B/é�.?1ÿЧXÚ\�ö!�m�ݶ3¤¦�éõéuª?nÃã�±��nØþ�)º§~X3�©�VSM¥��­�½§3VÇL«¥§KrÙ�ĤVÇ©Ù��O£²w¤"¦+��VÇÿЧpLSAE,Ø´��“���”��d�.§�Ï´3pLSA¥§éuéu©�→ÌK�©Ù£=¤ÚÌK→c�©Ù£=¤E,w�´ëê Ø´�ÅCþ"ÌK�.§=topic models§3 [11]1�g�wªJÑ5§¢Sþùp�ÌKÒ´���cd3�Â

©Û¥��Â�Ý"3ùp§·�3ÌK�.�µe�[/e0��e�o´ÌK£topic¤"ÌK´�1���2011c6�6F§�â�y|¢ÚO"2http://lucene.apache.org/3éuù��.kpLSIÚpLSAü«�{§Ù¥pLSI¥�I´“indexing” �"Ï��@LSI´3u¢�µeJÑ�§¤±pLSI÷^

�c��{"�´�Yó��mЧ٢pLSI®²Ø==ØÛ�uu¢¯K§¤±pLSA��Æ�"

1

Page 3: ÌÌÌKKK˝˝˝333'''˛˛˛ ÷÷÷¥¥¥˙˙˙AAA^^^net.pku.edu.cn/~zhaoxin/Topic-model-xin-zhao-wayne.pdfComputer Networks and Distributed Systems Lab Peking University Beijing, China

Figure 1: A�ÌK�~f"

�8Ü�6�§�Ò´`�½ØÓ��§§���Ûõ��´ØÓ�"ÌK´��8Üþ�Â�pÝÄ�!Ø L«"ã 1�ÑA�ÌK�~f§·��±w�z��ÌKéA��'�����Â"3ÌK�.¥§z�ÌK�L«¤��õ�ª©Ù§3¢SA^¥§·� ��c10�'�i��(JЫ"z�ÌK�é©�L��SN�\Ä�!�\Ø §ÏLã 1§·��±uyz�ÌK´éu©�SN�d3�ª��÷"

duÌK�.�ûÐ�êÆÄ:Ú(¹ÿÐ5§�²íѧêþ��5g��+�Æö�'5§,��2�/A^�«©��÷Ú&E?n�?Ö¥"du8cÚÌK�.A^�'�Ø©k�VC3000�§·�ØU¡Þz�Ø©§�©òÌ��7±e?Ö?1Ðmµ'X�äêâ�÷!�a©Û!?z©�6©Û!�EØ©�÷!�¬xN�÷±9�DÚ�&Eu¢?Ö�kg,�ó?n¥��?Ö"éu�{nØÜ©§·�ò¦þ�Qã�Ø%¯K§¿��Ñ�¡ q H�í�[!"

2 ÌÌÌKKK���...���{{{VVVããã

3�Ù§·�òéuÌK�.��'��{Ú�£?10�§ùéun)�YÙ!�SNò¬å��½��Ï�^"

2.1 ÄÄÄ:::���£££

·�Äk0�±eVgÚ�{µ

• ÚO�ó�."Äk¤õ|^êÆ�{)ûg,�ó?n¯K�´�ÑÚ�ó?n��_pZ�(Fred Jelinek)"��_pZ�3IBMúi�Æâ>b(Sabbatical Leave)§+��1#Ñ��Æ[|^�.O�Å5?n<a�ó¯K"ÚO�ó�.Ò´3@��ÿJÑ�"ÚO�ó�.´ïÄ��©�S��)¤VÇ�¯K"~X§�½��S�S = w1, w2, ..., wL§·����O½öO�Pr(S)"

• N��ó�."ÚO�ó�.F"éu���½�S�UÑÑ��Vǧ=Pr(S)"���~���.Ò´N��ó�.§=Pr(S) = P (w1)P (w2|w1)...P (wi|wi−1, wi−2, ..., wi−N+1)...P (wL|wL−1, wL−2, ..., wL−N+1)"nØþ§�XN�O\§�.�LyUåòØäO\§�´Ó��.�E,Ý�Ó�O\§3¢SA^¥§ É�u���k�5"¤±8c�~^�N§E,´1!2Ú3"�N = 1§ù��.��OC¤Pr(S) =

∏Li=1 P (wi)"���.duÙ{üÚk�

5��4�2�/A^"Ù¥ã 2�Ñ�����ó�.�VÇãL«�{"���ó�.��L«¤��õ�ª©Ù"

• ��·Ü�ó�."~��·Ü�.Xpd·Ü�.§3pd·Ü�.¥§z�|�´��pd©Ù"3��·Ü�ó�.¥§z�|�´�����ó�."�½��©�8ܧ·�Äkb

2

Page 4: ÌÌÌKKK˝˝˝333'''˛˛˛ ÷÷÷¥¥¥˙˙˙AAA^^^net.pku.edu.cn/~zhaoxin/Topic-model-xin-zhao-wayne.pdfComputer Networks and Distributed Systems Lab Peking University Beijing, China

�§�éAK�æ§,�������©©©���888ÜÜÜkkk������éééAAAK���æææ���©©©ÙÙÙ§,��âù�©Ù5ÀJ��æ§z��æéA�����ó�."�À½æ�§,�dTæéA����ó�.)¤éA�©�"ã 3�Ñ����·Ü�ó�.�«¿ã"

• pLSI�. [37]"pLSI´éud3�©Û�VÇÿÐ"§b���©�8ÜéAkK�ÌK§,�zzz���©©©���kkk������©©©���AAA½½½���éééuuuK���ÌÌÌKKK���©©©ÙÙÙ§©�¥�z�c)¤©�üÚµa) Äk�â©��ÌK©ÙÀJ��ÌK¶b),�dTÌKéA����ó�.)¤Tc"ã 4�Ñ��pLSI��ó�.�«¿ã"

• VÇã�. [5]"VÇã�.´�«òVÇÚDÚ�ã(�(Üå5�L«�{"ÏLã(���ßL«§·��±�Ð/n)VÇ�©Ù§VÇã�.´���~nØ�\ qSN2��Æ�§3d·�==ùã�éun)ÌK�.k�Ï�SN"ã 5Ы��LDA�VÇã�.�L«�{"Ù¥��L«Cþ§XJ�æ¤�ÚKL«TCþ®�§ÄKL«TCþ��"Cþm��Þ��L«VÇ��'X"Ù¥��µL«SÜ(��­E§�µme���IL«­E�gê"¢Sþ·��±uyLDANyÑ�«�?'X§§´�«�?��d�."

Figure 2: ���ó�.«¿ã"

Figure 3: ��·Ü�ó�.«¿ã"

Figure 4: pLSA«¿ã"

e¡·�5'�±eã 2, 3Ú4µ1¤���ó�.b���©�8Ü=k���ó�.¶2¤��·Ü�.b���©�8ÜkK����ó�.§�´�k���Û�VÇ©Ù^5ÀJùK��ó�.�,��¶3¤pLSIb���©�kK��ó�.§éuz��©�k��©�A½�VÇ©Ù�±�^5ÀJùK��ó�.�,��"ùn«�.Ny<�é�©��óï�L§�&¢ÚØäO\�.Lyå�L§"¢Sþ§LDA==´pLSI�����d�ÿЧÙ)¤L§�pLSI��þ´�Ó�"

2.2 LDA���{{{���000���LDA [11]´Latent Dirichlet Allocation�{¡§´�«�g���d�."��B±��Q㧷�e¡�[/0��/ªz�Ä:"·�b���©�8Ü��kT�ÌK§z�ÌKz�L«¤��c;Vþ����ó�.θz§=c;þ���õ�ª©Ù"·�?�Úb�z�©�déAùT�ÌKk��©�A½�õ�ª©Ùφd"��©��)¤L§Xeµ

• æ�φd ∼ Dir(α)¶

• éu©�d¥�z��cw§·�µ

Figure 5: LDA�ã�.L«"

3

Page 5: ÌÌÌKKK˝˝˝333'''˛˛˛ ÷÷÷¥¥¥˙˙˙AAA^^^net.pku.edu.cn/~zhaoxin/Topic-model-xin-zhao-wayne.pdfComputer Networks and Distributed Systems Lab Peking University Beijing, China

– æ���ÌKI\z ∼ Multi(φd)¶

– )¤éA�w ∼ Multi(θz)"

Ù¥θ(·) ∼ Dir(β)"3ÌK�.¥§·�I�'5ü«Vǵ1¤�«´©�∼ÌK�©Ù¶2¤�«´ÌK∼c®�©Ù"ÄuùÄ:§·�e¡?Ø�en)LDA��­��£µ

• ��d�?�."pLSIÚLDA���«OÒ´LDA´“���”��d�.§ pLSIØ´"3pLSI¥§φ(·)Úθ(·)Ñ´ëꧦ�¬�X©��ê8O\ÚÌKê8�O\ O\§ 3LDA¥§Krφ(·)Úθ(·)w��ÅCþ§¦��´5guéA��ëê����ÅCþ"

• ���5"���5��´§�½��k��Ý�CþS�z1, z2, ..., zN§éuTS��?Û����π§kP (z1, ..., zN ) = P (zπ(1), ..., zπ(N))"XJ��S�¥�Cþ÷vÕá�Ó©Ù(i.i.d.)§@oTS��½´����§�´���7¤á"�âDe Finetti½n4§���5��¢S´^���Õá�Ó©Ù§=�½û½ùCþ©Ù�ëê±9©Ù¼ê�§ùCþ�©Ùâ÷vÕá�Ó©Ù"

• Û¹Cþ�."LDA��þ´�«Û¹Cþ�.§ù�´¶i¥�¹�“Latent”Ì�¿g"Û¹�.��´§Ú\�¥mÛ¹CþXÌKI\§ùCþ3¢Sê⥴JÃ�§¿�Ø��"Ú\Û¹Cþ�Ì�8�´¦��.�£ãÚí��\�ß!{ü"

• VÇ�Ý"�â��d½nP (θ|X ) ∝ P (X|θ)P (θ),VÇ�Ý��´��VÇP (θ|X )Úk�VÇP (θ)kXÓ��VÇ/ª§ d�P (X|θ)�VÇ/ªÚP (θ)�VÇ/ªÒ�����Ýé"3LDA¥§Dirichlet©ÙÚõ�ª©ÙÒ´���Ýé"

2.2.1 kkk������½½½

3LDA¥§kü|k�§�«´©�∼ÌK�k�§5gu��é¡�Dir(α)¶�«´ÌK∼c®�k�§5gu��é¡�Dir(β)" [30]�Ñ�²�5α, β���{§Ù¥α = 50.0

|T | , β = 0.1" [29]y²¢SþpLSI´�«LDA�.�MAP£maximum a posteriori¤�O§�k�æ^�´é¡�DirichletVÇ"¢SþduLDAæ^�������d廧éu��©�!c®��O�\O(§Ó�pLSI��±ÏLæ^MAP��O�{5Ú\k�"3���©��÷?Ö¥§ùü«�.�¢S�JAT�C§�´LDAw��\(¹!nØÄ:�j¢§AO´��Ä�©��÷¯KAOE,§LDA�\N´¢y(Üõ��.|�3���.¥"

[89]�Äùü«k���½�ªµ1¤é¡�¶2¤�é¡�"(Üü«k��ü«ØÓ�k��½�{§·��±��±eo«|ܵ

• AAµ©�∼ÌK©ÙÚÌK∼c®©ÙÑæ^�é¡�k�¶

• ASµ©�∼ÌK©Ùæ^�é¡�k�§ ÌK∼c®©Ùæ^é¡�k�¶

• SAµ©�∼ÌK©Ùæ^é¡�k�§ ÌK∼c®©Ùæ^�é¡�k�¶

• SSµ©�∼ÌK©ÙÚÌK∼c®©ÙÑæ^é¡�k�"

¢�uyæ^AS��ª�±�Ð/JpLDAéu©�ï��Uå"d�k��ëê��d��{Ú\�E,�k�&E§~XDirichlet Process [83]ÚPitman-Yor

process [85]§·�ò¬3�YÙ!���'SN�2�[0�"

2.2.2 ���...¦¦¦)))

éuIO�LDA§�.¦)´���~E,��`z¯K§éJk°(¦)��{"¤±·�æ^Ø°(��.¦)�{§�Vkn«Ø°(��{?1�.�¦)µ�«ÄuGibbsæ���{ [30, 34]§,�«´ÄuC©{�EM¦) [11]§�k�«´ÄuÏ"í?��{ [62]"��5`§Gibbsæ���{í�å5��{ü �¦)�J�Ø�§¤±3�©¥§·�òæ^Ù��Ì��{?10�"·�Äk{�0��eGibbs Sampling"Gibbs sampling´��Metropolis-Hastings�{5 [5]�A~"ÙÄ�g�´§�½��õ�Cþ�©Ù§�'uéuéÜ©ÙÈ©§l^�©Ù¥æ��{ü"b�·���l��éÜ©ÙVÇP (x0, x1, ..., xn)¥¼�K���X = {x0, x1, ..., xn}"T�{�ü�Ï^Ú½µ

4http://en.wikipedia.org/wiki/De Finetti’s theorem5http://en.wikipedia.org/wiki/MetropolisõHastings algorithm

4

Page 6: ÌÌÌKKK˝˝˝333'''˛˛˛ ÷÷÷¥¥¥˙˙˙AAA^^^net.pku.edu.cn/~zhaoxin/Topic-model-xin-zhao-wayne.pdfComputer Networks and Distributed Systems Lab Peking University Beijing, China

• �Å/Щzz�Cþ¼�X(0)¶

• éuz���X(i), i = 1, ...,K§éuz���Cþxj§·�l^�©ÙVÇP (x(i)j |x(i)

1 , ..., x(i)j−1, x

(i−1)i+1 , ..., x

(i−1)n )

��x(i)j "

3LDA�ÄuGibbsæ���.¦)¥§ æ^“Collapsed Gibbs Sampling”�{ [30]"Ä�g�Ò´Ø�Äθ(·)Úφ(·)ü|�ÅCþ§ ��Äéuz�c�ÌKI\z�ín¦)"ù¦�O�Úí���{z"·�3N¹¥�Ñ���~�[�í�ø�[ÆS"3¢SA^¥§du©�êþ�ã�!©�6��SA�§k�ïÄm©'5LDA�¯�ín�

{ [72]!3�ÆS [36]!©�6�ín�{ [101, 4]!©ÙªÆS [65, 49, 100, 3]"ùïÄò¬¦�éuLDA�.¦)��Ç����Uõ§Ó�¬ò·A©���SA�§�ÎÜ¢S©�6�A�"

2.2.3 ���...µµµ���

éuÌK�.�µ�´����±5��'5§�´¿��éÐ)û�¯K"Ì��ϧ·@�´duLDA��´�«©�L«�{§ ��éJµ���L«�{�Ð�"8ck��{§�V�±©�±enaµ

• ÄuPerplexity��{"Perplexity²~�^3�ó�.¥§´^5ïþ�ó�.éuÿÁ���ï�Uå�“Ð�”§=q,���"/ªz§�±�âXe�ªfO�ÿÁ8Üþ�Perplexity���

perplexity(Dtest) = exp{−∑

d log(P (wd))∑d Nd

} (1)

���#�ÌK�.�JÑ�§ ÏLÚIO�LDA?1ÿÁ8Üþ¡�Perplexity�é'§XJ�����perplexity§Ò@�d�.�ï��J�Ð�"�´ÄuPerplexity'�������¯KÒ´§���±¼��$Perplexity��.�7�½�±ÆS��<�@�éÐ�ÌKc"

• ÄupVÇÌKc"z��ÌK�ª�L«/ª´�����ó�.§·��±�âz�ÌKSÜc®VÇ�p$5?1ÌKc®�üS"���ùpVÇ�ÌKc® �±����ÑÑЫ�^r§4^r5?1µ�" [24]1�gXÚ/�ï�ó�.�<óµÿ�{"Ì��Äü��ݵ1���Ý´ÌKSÜ���5§äN�{Ò´éu��ÌK§ÄkÀJäk�pVÇ�5�ÌKc§,��Å/V\þ��3�cÌKekX�$VÇ�´3Ù¦ÌKSÜäk�pVÇ�c®¶1���Ý´©�SÜÌK©Ù���5§äN�{Ò´éu��©�§·�ÄkO���VÇ�p�n�ÌK§,�·��Å/V\Ù¦��ÌK"éuùü�?֧Ѵ<ó/µ�u�Ñ�ÅV\�c½öÌK�J´§Ý" [90]­#u��µ�ÌK�.��{§uy¢Sþ�c��{´ØO(5µ�ÌK�.�§qJÑü«#�µ��{" [19]Äu [90]�ó�q­#?1ÿЧJÑ#��OÿÁ©�8Ü�q,��{" [1]JÑéuÓ��ÌK�.ÆS���ÌK?1­�ÝüS§Ä�g�´Ø´¤kÆS���ÌKÑ´�Ó�§ÌKüS�±gÄ/uy�\“­�”�ÌK"

• |^Ù¦?Ö��J5m�µ�"éu���ó�.§·��±ÏL�.¦)§�ª�±��ü«VÇ1�«Ò´©�∼ÌK�Vǧ1�«Ò´ÌK∼c®�VÇ"ùü«VÇÑ�±3�Ù¦?Ö¥¦^§X©��qÝ�O�§ÌKm�qÝ�O���"ÏLùm�?Ö5'�ÌK�.�Ð�k��¯KÒ´§éuØÓ�?Ö§z�ÌK�.�`:�UØ��§¤± ��?ÖØUïþÑ5ü�ÌK�.�m�ØÐØ�"ù«�{'�Ü·3��A½?Ö¥JÑ�ÌK�.�Ð�§Ý§X�a©a"

2.2.4 ÌÌÌKKKêêê888���(((½½½

éuÄuLDA�ÌN�.§���~­��¯KÒ´XÛ(½ÌKê8" ÌKê8�(½¢Sþ´���.ÀJ��.§Ò��c�ff?ØL��.µ�Ü©§du�.µ���´���~(J�¯K§¤±éuÌK�.¥��.ÀJ¯KE,´���~(J�¯K"8c��{�Vkü«µ

• ²��½"3�©��÷ó�§ïÄ< ÏL�E/NÁ½öqÞÌK�ê85*¢��J�Ð�§~X*pVÇ�ÌKc®�Ð�!�´Ä����"ù«�{�,´éuª

5

Page 7: ÌÌÌKKK˝˝˝333'''˛˛˛ ÷÷÷¥¥¥˙˙˙AAA^^^net.pku.edu.cn/~zhaoxin/Topic-model-xin-zhao-wayne.pdfComputer Networks and Distributed Systems Lab Peking University Beijing, China

Figure 6: Supervised-LDA�ã�.L«"

Figure 7: Labeled-LDA�ã�.L«"

�§�´ ¢S¥{ü´1§Ï ´�~^��{"¢Sþ§éu�Ü©�ïÄó�Ñ%@vkéÐ�½ÌKê8��{§ ��æ^ù«²�5��{5�½ÌKê8"

• ÄuPerplexity�(½�{"3�.µ�¥§·�ù�XJ��ÌK�.3ÿÁý�8Üþ¼��$�Perplexity§@o·�Ò@�ù�ÌK�.äk�Ð��.L«Uå"�´�´@�P¯K§Perplexity���ÚÌK�.3¢S¥?Ö¥�Ð�3nØþ¿ØUk���öS"¤±3�äN�©��÷ó�§ù«�{¿Ø�æ^§ ´��¦^²��½��{¶�´éuÅìÆS�«§ ÏLd�{5�y��#JÑ�ÌK�.�Ð�"

• ¦^�ëê���d�{éÌN�.?1ÿÐ"·�ò3e�!éuù��{?1�[/0�"

2.2.5 ÄÄÄuuuLDA���ÌÌÌKKK���...CCC///

�XLDA�íѧ���8U§�yÆâ|¢�w«�Ú^Ø©êp�C2900g"�1�ÆöéuÄ��LDA?1�«C/ÚÿЧ�k3�«?Öþ�A^"3d§·�Áãòù�.�CzÚÿÐ?1��oÑ�o(§/d5�Ð/)ÌK�.�uÐ"

• �»�k�����b�"3�©�LDA�.¥§����g�A^�ü�/�µ1¤©�8ÜSܧ©��m´����¶2¤©�Sܧc�c�m´����"�´ù���b��´3ï�L§¥�~��.�E,Ý �á�§¢Sþùb��½�.�L«Uå"�dk�

Figure 8: Dirichlet-multinomial Regression (DMR) topic model�ã�.L«"

6

Page 8: ÌÌÌKKK˝˝˝333'''˛˛˛ ÷÷÷¥¥¥˙˙˙AAA^^^net.pku.edu.cn/~zhaoxin/Topic-model-xin-zhao-wayne.pdfComputer Networks and Distributed Systems Lab Peking University Beijing, China

�.? éuù���b�?1tµµX [9, 46]Ú\ÌK�m�'é! [23]Ú\©��m�'é! [96]�Äc�c�m�^S'X"�´���»�k����5��§�.�E,ÝòwÍO\§¤±7L�Ä���¯K´§O\ùE,Ý�§���#ÌK�.UÄ3¢Sêâþ¼��Ð��Ç",��ò¥��{§´3�O�.L§§·�¿Ø�»ù���b�§ ´��3`zúªÄ:þO\�U?"XO\k�&E5«OØÓ�ÌK [51]!O\5KzÏf [56]5Ú\�'é&EÚk�&E"

• Äu�ëê��d�{�C/"Ù¥'�k�L5�Ò´ÄuDirichlet Process [83]�C/"ÄuDirichlet Processd��{�±gÄ/ÆSÑ5ÌK�ê8 [84]"@oÒkù���¯K§´Ø´ÄuDirichlet Process��{Ò�±)ûÌK�.¥�gÄ(½ÌKê8ù�¯Kº�Y´ü¡�µ1¤3�½§Ýþ)ûÌK�.¥gÄ(½ÌKê8ù�¯K¶2¤�d´·�7L��%/��½!N��Ù¦ëê�ê�§~X�ëê��½¶3¤ÄuDirichlet Process��{ 3¢S¥$1E,Ý�p§du3ÆSL§¥okÌK��Ú�)§ùò¦��èC�E,ÚJ±�o"¤±3¢S¥§ ���ò¥§wwgÄ(½ÌKê8ù�¯K�.éu��A^¯K�I¦�.kõî�§XJ²��½Ò�±÷v�{§@oÒØ^æ^Äu�ëê��{¶�´éu�du¯K§X�Ú\�k��£½ö(�z&E§3ù«�¹e§ �ëê��{´`kÀJ§~XäG�g�ÌK�. [7]Úk�Ã�ã(��ÌK�. [46]"�k�Ù¦�ëê���d�{§XÄuPitman-Yor Process��{ [85, 79]"

• lÃ(�&E→(�z½ö�(�z�&E"IO�ÌK�.´�«ÃiÒÆS��{§�I�Ñ\ÌKê8Ú��©�8Ü�¤k©�§�.ÒU?1ÌK�gÄÆS"éu�A½�A^¯K§~X©�©a§�·�®²kÜ©Ôöêâ�£~X§©��aOI\��¤§@oXÛ3ÌK�.¥XÛ¦^ùÔöêâ�&E"XJ2rù�êâ��¦?1tµ§@oÒ´�ÄXÛK\©�N\&E"X©� r©���w����c�f§Øc�f§vk?ÛN\&E"�X©�êâ�ª�´LÚpé�êâ�uЧDÚ�X©�*: طܧN´�Ñ�é­��Ù¦A�§~X�mI\!aOI\!^rJø�I\��"¤±3ÌK�.¥§���~­����Ò´XÛK\ÌN�.¥�ùk^�A�"3¤kùA�¥§�ö¢N!�m!�ä(�!I\&E��Ñ´�~;.�A�§��Æö��2�'5"ù«&E�K\�±w�,«iÒÆS½öfiÒÆS��{§¢Sþ´�«XÛ3ÌK�.¥K\(�z�&E" [10]JÑÓ�'éÌKÆSÚ�ACþ£=�§response variable§§�±´^réu��©��µ©!©��aOI\!½öÙ¦ê�Cþ¤�)¤"Xã 6¤«§ÙÄ��{Ò´'éåz�©���ACþÚéAT©��c®�ÌKI\�þ"ù«'éå�&EÚÌK��{3���ó��²~¦^§~X'é�mÚÌK",���'ó� [75] K´��ïáålÑ�I\�ÌK�m�N�'X§éuz�I\§·��½Ùk��5/N����Ü©�ÌK§ Ø´¦^�cz�©�∼ÌK�þ!k�©Ù" [60]´��'�²;�ó�§?�Úò [75]��{�õ"Xã 8¤«§§ÏLÚ\��log-lineark�3©�∼ÌK©Ùþ¡§�±òÌKÄ�'é�©��“Metadata”��õ«A�£~X�ö!�m��¤§XJ�·�==�Ä�ACþ��T©��A��§¢Sþ [10]�±w� [75]���A~"

éAupLSA§�·�Pk�(�z�k�&E§·��±ÏL¦^Dirichletk� [51]5\\ù(�z���§½ö·��±ÏL¦^5Kz��{éu8I¼ê�\�� [56]"éu1�«�{§ÄuLDA��.Ó��±¦^§¢Sþ·�3�c®²QãLù�¯K§Äu��d�{�ÌK�.�±�(¹/¦^õ«õ��k�&E"éu1���{§du5KzÏf�Ñy§·�I�¦^C©{�EM�{?1¦) [99, 80]§ Gibbsæ���{ éJ?nù«5KzÏf§I��éuª��{ [82]"¢SþéuLDA§�·�kvk�&E5ín��c®�ÌKI\§·��±3Gibbsæ�¥���½Tc�ÌKI\ Ø?1I\�æ� [47]"

3 ÄÄÄuuuÌÌÌKKK���...���AAA^

duÌK�.��{A��A^�¤k�©��÷Ú&E?n+�§3d·�==�[/£��eA�É'5AOõ�+�µ�a©Û!��©�6©Û!�¬xN©Û!Æâ©Ù�÷!�ä(�z�êâ�÷Ú�Ù¦A^"

3.1 ÌÌÌKKK���...333���aaa©©©ÛÛÛ¥¥¥���AAA^

�a©Û [69]£=��opinion mining!sentiment analysis¤´Cc5#Ñy���ïÄ��"ÙÄ�?ÖÒ´l^r)¤��¹*:Ú¿��©�¥Ä�Ñù*:Ú¿�§,�)¤�aÁ�!?1�a

7

Page 9: ÌÌÌKKK˝˝˝333'''˛˛˛ ÷÷÷¥¥¥˙˙˙AAA^^^net.pku.edu.cn/~zhaoxin/Topic-model-xin-zhao-wayne.pdfComputer Networks and Distributed Systems Lab Peking University Beijing, China

©a!gÄ�ï�ac;���a©Û?Ö"��þ5ù§�a©Û´��A^£½ö`��¤°Ä�)�#��"�X�ä�uЧ<��±é�B/3�ï�«�¬!ÑÖ§Ó�q�±��uLgCéuù�¬½öÑÖ�*:ÚïƧ ���^r q¬ë��c�^r�µØ§± E©§ù«�¤�ª±9^r)¤�êâ¦��a©Ûéuó�.!û�.�­�§éu^r���´���~k¿Â�©��÷?Ö"ÌK�.3�a©Û�Ì��?ÖÒ´ÆSÑ5^r?Ø!^rµØ¥�SNÌK"3�a©Û

¥§z��topicÏ~�¡�aspect§�Qã��BÚÚ�§3d·�ò÷^ÌK5��aspect��È"3�a©Û©Û¥§¬òc®«O¤��ac®£Xgreat, good��¤ÚÌKc®£Xfood, drink��¤"éu�3�Ø��Õ§ k�ý���“ratable”�ÌK§¿�N���©Å�§X0-1�©£÷¿!Ø÷¿¤§©?�©£1∼3½ö1∼5¤"ùp�Qã��B§E,÷^“ratable”ù�=�üc§Ø?1�È"l^r)¤�µØêâ¥J�ÌK�V©�n��{:

• ÃiÒÌKÄ�"IO�ÌK�.XpLSI [37]ÚLDA [11]Ñ�±��A^�^rµØ¥§?1ÃiÒ�ÌKÄ�"�´ [87]uyDÚ��ó�.XLDAl^rµØ¥ÆS���ÌK���éA,�¬ý Ø´éA�“ratable”�ÌK" [87]©Û���~­���ÏÒ´�c�ÌK�.�Ä�´©�SÜ��y§Äu©�ù�“��µ”(context)ÆS���ÌK ´éA¬ý Ø´�±“ratable”�ÌK"�)ûù�¯K§ [87]�Ä��õ?��µÌN�.µc∼éf∼ãá∼©�§ÏLù�²LÁ��y§ù«õâÝ�ÌKï��{�±�Ð/Ä�“ratable”�ÌK"�Y��ó��?�Ú÷^ù«g�§�´¦^�é��{ü��.µ1¤rü�éf����©�§,�¦^IO�LDA [17]¶2¤éu��éfæ���ÌKI\§ Ø´éu��cæ���ÌKI\ [40, 44]"ùü«�{�g´Ñ´M5/ �“�µ”§�´3¢SA^¥��Ø���J"

• fiÒÆS"38c�3�µØ�Õ¥§Ø^r�©iµØ§ kéõ(�z½ö�(�z�&E§~X^réu��û¬��©!^réuû¬V\��I\"3�cÄu^r)¤êâ¥Ä�ÌK�ó�¥§�����'�¯KÒ´XÛòÆS�����ÌKXéA�¢S¥éu©Û^r�ak�Ï�ÌKY"�ïÄó�<|^3�µØ�Õ¥(�zÚ�(�z�&E5�ÏÄ��äk¢S¿Â�ÌK"äN�{�Vk±eü«µ

– |^�.�k�&E"ÃØ3pLSI¥�´3LDA¥§·�Ñ�±ÏL�½k�&E��{5�ÏÄ�ÌK"ÙÄ�g�´§éu�ÌK§·�k�'�O(�k�&E§~X§éu�ð��¬§·��±l�Äz�¥Jѧ���A��£ã§,�ò§�Ôö��k�&E"~X/Ï�Äz�¥(�z�©�5�ÏÄ�Æ�¥�ÌK [51]"

– K\(�zA��ÌK�.¥",�«�{§´·���{r�äk(�z&E�A�K\�ÌK�.¥"äN5`§·�Ó�'éü�)¤L§§��Ò´©�¥c�)¤§,��Ò´ù(�zA��)¤"ÏLù«'駷��±m�/Ú\ù(�z&E�ÌKÆSL§¥ [86, 16]"

• éÜÄ�ÌKÚ*:"�c��{�´lÄ�ÌK��Ý��Ä�§3z�ÌKSܧ ¬rÌKcÚ�ac·Ü3�å§Ø\«©"k�Ü©ó��Ä«©m�aÚÌKü«ØÓa.�c®§? Ó�Ä�ÌKÚ*:"ùó��V�±k±eü�廵

– éu¤kÌK§�½�X��k��a�ó�.£Ï~�)�Â!@ÂÚ¥5n«�aI\¤ [57]"�Ò´`3©�¥�z�cÄkæ�����Cþ§5û½´ÌKc®�´�ac®"XJ´ÌKc®§���)¤L§Ú3IO��ó�.¥��¶XJ´�ac®�{§Òl�a�ó�.¥ÀJ��Ü·��aI\"Ù¥ã 9

– éuz�ÌK§�½��A½��a�.§3ù«�¹e§ Ø2[©�ó�.�45§ �´ÆS����ÌKA½��ac��ó�."

3éÜÄ�ÌKÚ*:��ÿ§�J�¯KÒ´XÛ£O�ac®§�é{`§XÛ©mÌKc®Ú�ac®"1�«�{Ò´|^�®²k��ac;��k�&E§,�3ÌK�.ÔöL§¥2uy#��ac®¶,�«�{Ò´Ú\�iÒÆS�|��ÌK�.¥§ù�·��±Ó�okiÒÆSÚÌK�.�`:µéuiÒÆS5`§§'�·Ü«©ÌKÚ�ac®§�´Ø·Ü^5àa�Â�C�c®¶éuÌK�.5`§§'�·Üàa�Â�C�c®§�´��ÃiÒ�ÌN�.Ø·Ü^5«©ÌKc®Ú�ac®" [107]ÏL(Ü���|�ÚÌK�.§¦�T�.Ó�äkiÒÆSÚÌN�.�`:"

8

Page 10: ÌÌÌKKK˝˝˝333'''˛˛˛ ÷÷÷¥¥¥˙˙˙AAA^^^net.pku.edu.cn/~zhaoxin/Topic-model-xin-zhao-wayne.pdfComputer Networks and Distributed Systems Lab Peking University Beijing, China

Figure 9: ��ÌK-�aéÜ�.�«¿ã [57]"

Figure 10: ��ÄuÌK�Á�«¿ã [107]"

�·�Ä�ÌKÚ�ac®��§·�Ï~ �±|^ÌK�.�ÑÑ5?1µ

• ©��a©a"·��±O�z�©��N��a�� [57, 47]§du3ÌK�.¥§ÃØ´ÌK�´�a§Ï~Ñ´L«¤���ó�.§Ïd·��±¦^IO�VÇ�{5O���©���a��§½ö·���3�.¥�½�aI\ [57, 47]"�k��.�½©�AÏ��aI\½öcA½��aI\§�N´5�¤ù�?Ö"

• )¤ÄuÌK��aÁ�"ÌK�.�(J�±��^����a��8Ü�Á�§8c�Ü©�ó�Ñ´)¤ÄuÌK��aÁ�§�Ò´UìÌK|��ac®ÚÌKc®" [107]?�ÚJѦ^ÌKAÏ��ac®?1�aÁ�"ÌKAÏ��ac®�ÌKc®�mk�r��Â'é"ã 10�Ñ��ÄuÌK��aÁ�"

• gÄ�ï�ac;"ÆS����ac®�±���ac;�ÿÀ§·���±?�ÚN\þÌK�µ&E5´Lù�c;�L�Uå"

3.2 ÆÆÆâââ©©©ÙÙÙ���÷÷÷

Æâ©Ù�÷´ÌK�.���­�A^§ÏLéuÆâ©Ù��÷5?�Ún)Æâ©Ù�uÐ!?z§ùéu)�c��E?Ú!�5��EuЪ³Ñ´�~k¿Â�"·�3déuùïÄ?1��oÑ�©a"

3.2.1 éééuuu���ööö???111ïïï���

Author-Topic Model [77]´l�ö��Ý5�Ä©�¥ÌK�)¤�"

9

Page 11: ÌÌÌKKK˝˝˝333'''˛˛˛ ÷÷÷¥¥¥˙˙˙AAA^^^net.pku.edu.cn/~zhaoxin/Topic-model-xin-zhao-wayne.pdfComputer Networks and Distributed Systems Lab Peking University Beijing, China

Figure 11: A«Ú�ö�'�ÌK�.«¿ã"

Figure 12: Pairwise Citation LDA model«¿ã [64]"

éuÆâ©Ù��8ܧÌK�. [11]¿vk�Ä�©Ù��ö§¢Sþ´r¤k��öÑw����Ó�¶ü�ÌK��ö�. [53]§?Û���öÑéAX��A½��ó�.§�´ù¢Sþ´�«�~r����¶¢SþAuthor-Topic Model [77]´éþãü«�.?1�ò¥�Ä"éuz���öØ2�½T�ö�UéA��ÌK§ ´z��öéAu��ÌKþ�©Ù§Ó�©�∼ÌK�©Ù����§=��ö∼ÌK�©Ù � ��¶Ó�§·��±w�¤k��ö����ÌK�8Ü"

[54] ?�ÚÿÐAuthor-Topic Model§ÏLÚ\�Âö��§=?Û©�?Û��cÑ´d��£�ö§�Âö¤é�Óû½�"�c3´z��öéA��ÌKþ�õ�ª©Ù§y3���ö§�Âö¤ééA��ÌKþ�õ�ª©Ù"ã 11é'Ыo«�.§3¢SA^¥§Author-Topic Model [77]´�~¦^�Ó�éu�öÚÌ

K?1��{"

[66]l,���Ý5�Ä“�ö”��Ú§=Ø©µ"ö��ݧ3ù�ó�§�öb�z��ö´k�J[�“�5z”ÌK©Ù§ Ø´�Author-Topic Model¥§z��ö==k��ÌK©Ù"

3.2.2 éééuuuÆÆÆâââÚÚÚ^ïïï���

Æâ©ÙØÓ���#ª��!Æ�©Ù��§��ÑI�N��~�[�ë�©z§ùë�©z Ñ��©�Ø©kX�~;��éX"¢Sþù´�wª�Ø©�Ø©�m�éX"XÛéuùó�'X?1ï�§��´Æâ©ÙïÄ���9:Ú'�¯K"

[64]´1��JÑ5Ó�éuÌKÚë�Ú^?1ï��©Ù"ÙÄ�g�Ò´Äk©Oéu©�æ^�cIO�ÌK�.�)¤�ª§~XLDA [11]§,�éu?Û�éäkÚ^'X�©�é§�â©�∼ÌK©Ù��q5)¤Ú^ó�'X"ù«?n�ªò©�Úó��m�'éÏLÌK©Ùïá

10

Page 12: ÌÌÌKKK˝˝˝333'''˛˛˛ ÷÷÷¥¥¥˙˙˙AAA^^^net.pku.edu.cn/~zhaoxin/Topic-model-xin-zhao-wayne.pdfComputer Networks and Distributed Systems Lab Peking University Beijing, China

Figure 13: Citation Link-PLSA-LDA«¿ã [64]"

Figure 14: ØÓ�ÌK?zï�g´«¿ã [33]"

å5"ã 12�Ñù��.�«¿ã"ù��.�E,ÝAOp§¤± [64]q?�ÚJÑ��U?��.§?�Ú{z�Ú^©��©�)¤�ª§�ã 13"gÄÆS½öuyÆâ©Ù¥�ÌK?z´���~­��¯K§ [33]�ÄXÛÿÐIO

�LDA [11]5\\Ú^&E§5�Ð/?1ÌK?zï�"b��½���m:t§éAk��ÌK�mZt§ÌK?z�Ì�?ÖÒ´éuz���m:tÆS�ÑéA�ÌK�mZt§Ó�uy���m:mÌK�Cz§�kÓ��ÌK3ØÓ�m:S�Cz"Ù¥�Ì��¯KÒ´XÛ�OÑz�z���m:tÆS�ÑéA�ÌK�mZt" [33]?Øn«Ì��ÌK?zï�g´µ

• 1�«Ò´�mÕá�ÌK?z�ª"=z��m:�ÌK==��c�m:SÜ�©Ù8ÜDtk'X§ �±��vk'X¶

• 1�«Ò´�m\È�ÌK?z�ª"=z��m:�ÌK==��c�m:±9�c�m:�c�¤k©Ù8Ük'X¶

• 1�«Ò´Ú^¯a�ÌK?z�ª"=z��m:�ÌK==��c�m:©ÙDt9�c�ù

©ÙÚ^�©ÙLDtk'X"

[33]ÏL¢�uyù«ÄuÚ^��{�±�Ð/?1ÌK?z�ÆS"

3.2.3 éééuuuÆÆÆâââ���������©©©ÛÛÛ

ÄuLDA [11]éuÆâ���©Û´��ÌK�.3Æâ���÷¥²~?1�?Ö"Ù¥ [30]´�@���ÄuÌK�.5?1Æâ��©Û�ó�"§Ì�©Û@ÌK´“9”ÌK§=ÌK

11

Page 13: ÌÌÌKKK˝˝˝333'''˛˛˛ ÷÷÷¥¥¥˙˙˙AAA^^^net.pku.edu.cn/~zhaoxin/Topic-model-xin-zhao-wayne.pdfComputer Networks and Distributed Systems Lab Peking University Beijing, China

1. For each topic t = 1, . . . , T ,(a) draw φt ∼ Dir(β)

2. For each tweet s = 1, . . . , N ,(a) draw θs ∼ Dir(α)

3. for each tweet s = 1, . . . , Nu

(a) for each word n = 1, . . . , Nu,s

i. draw zu,s,n ∼ Multi(θs)

ii. draw wu,s,n ∼ Multi(φzu,s,n)

Figure 15: LDAµTweets�)¤L§.

´“e”ÌK§,�©ÛùÌK�X�m�uÐ�rÝCz§Ù¥Ì�¦^²þ�©�∼ÌK�©Ù5O�rÝ" [32]¦^LDA5©Û“ACL Anthology”6 l1978c�2006c�m���§Ì�©Û±eA�¯Kµ

• =ÌKC��\61"

• =ÌK9Ýeü"

• O��óƬƴC��5�A^íº

• ISn�Ì6g,�ó?n�¬ACL!EMNLPÚCOLING�m��É"

[109]©Û�öXÛK�ÌK?z�ù�¯K"�öÁã�£���¯K“�½��#ÌK§@où�#ÌKlÛ 5§�Ò´`l=ÌK?z 5Qº”" [109]@�ÌK�?z´�ö��ö�m��p �5�§JÑ��ê��Å�.5éuù«Äu�ö�p�{K�p?1ï�§¿�A^ù��.©Û�k¿g�¯K§Xé���½�ÌK?z§´=�öÌ�ùgCzQ"

[52] JѦ^ÌN�.?1[âÝ�©ÙK�å�©Û§Ì�´JÑ�O�Ýþ5éÆâïÄ?1©Û¶1¤Ú^ê8¶2¤ÌKK�Ïf¶3¤ÌKDÂÚõ�5¶4¤ÌK`k5¶5¤ÌKÆ·¶6¤ÌK[£"ÏLùÝþ330��m�Æâ©Ù��8Üþ�¦^§ [52]���k¿g�(Ø"

nÜþãùïħ·�uyÄuÌK�.��{éuÆâ��Jø�«[âÝ�©Û"Ì�©Û8¥3±eA��ݵ

• �m�Ý"ÌKXÛ?z!uÐ"z�ÌK�X�m�rÝXÛCz"

• SN�Ý"SN�ÝÌ�´�Ä�q5Úõ�5§=ÌK�m'��q§=©ÙCX�õÌK�SN"

• �ö�Ý"�öéuÌK�,�!�öXÛK�ÌK?z"

3.3 ���¬¬¬xxxNNN¥¥¥���ÌÌÌKKKJJJ���ïïïÄÄÄ

3.3.1 ÌÌÌKKK���...333Twitter¥¥¥���ÐÐÐÚÚÚ}}}ÁÁÁ

Tweets���«#.��¬xNêâ§�DÚ�©�8Ü£#ª��ÚÆâ©Ù¤kXwÍ�ØÓ§Ny3µDÑ�!�Ìá!�#�Ý4¯!êþã���"8c�Ü©ó�Ñ´ò®k�ÌK�.��A^3Tweetsþ½ö�\C/"Ù¥���J:Ò´©��Ý�á§�ckL�'ïÄ®²L²DÚ�ÌK�.£XLDA¤3á©�þ��JØÐ [51]"8c3TweetsêâþÌ�A^��.´LDA [11, 30]!Author-Topic Model£ATM¤ [77]ÚTwitter-

LDA [105]§ùn��.�~�q§�Ñk��g�ØÓ§ã 15, 16Ú 170�ùn��.3Tweetsþ�)¤L§7"

e¡·�5©Ûn«�.�m�éXÚØÓ"

6http://aclweb.org/anthology-new7�BuQ㧤k)¤L§Ñ^=�Qã"LDAÚATM�)¤L§Ú�©Ø©k�ɧùp·���Ñ3Tweetsþ�

äN)¤L§§ Ø´Ï^©�8Üþ�)¤L§

12

Page 14: ÌÌÌKKK˝˝˝333'''˛˛˛ ÷÷÷¥¥¥˙˙˙AAA^^^net.pku.edu.cn/~zhaoxin/Topic-model-xin-zhao-wayne.pdfComputer Networks and Distributed Systems Lab Peking University Beijing, China

1. For each topic t = 1, . . . , T ,(a) draw φt ∼ Dir(β)

2. For each user u = 1, . . . , U ,(a) draw θu ∼ Dir(α)

(b) for each tweet s = 1, . . . , Nu

i. for each word n = 1, . . . , Nu,s

A. draw zu,s,n ∼ Multi(θu)

B. draw wu,s,n ∼ Multi(φzu,s,n)

Figure 16: ATMµTweets�)¤L§.

1. Draw φB ∼ Dir(β), π ∼ Dir(γ)2. For each topic t = 1, . . . , T ,

(a) draw φt ∼ Dir(β)

3. For each user u = 1, . . . , U ,(a) draw θu ∼ Dir(α)

(b) for each tweet s = 1, . . . , Nu

i. draw zu,s ∼ Multi(θu)

ii. for each word n = 1, . . . , Nu,s

A. draw yu,s,n ∼ Multi(π)

B. draw wu,s,n ∼ Multi(φB) ifyu,s,n = 0 and wu,s,n ∼Multi(φzu,s) if yu,s,n = 1

Figure 17: Twitter-LDAµTweets�)¤L§"

• Äk§·��±uy¢SþATM´ÄkòTweetsàÜ/¤����^r/©�0§��þE,´��A^LDA8"�´ATMÏLàÜ�±é^r,�?1�xÚï�§ù�:´LDA��A^3ü�TweetþØU���§ATM�x�´3^r�¡��µ(½ö{ün)�/�y0)§ LDA�x�´K´Tweet�¡��µ"

• Ùg§ÏL'�ã 16Ú 17§·��±wѧTwitter-LDA¢Sþ´ÄuATM�Cz§Ì�kµ1¤Ú\�µ�.£ θB¤¶2¤z��TweetSÜ�¤kc®Pk��Ú��ÌKI\"

• Twitter-LDAÓ�éu^r�¡��µÚTweet�¡þ��µ?1ï�"z��TweetSÜ�¤kc®Pk��Ú��ÌKI\Ù¢´�«éM5�éuTweet�¡�µ?1ï���{§,�«�{´Ó�k^r�¡�ÌK©ÙÚTweet�¡þ�ÌK©Ù"�´duTweet�ÝLá§��ÃiÒ/ÆS§ék�UØU��n��TweetÌK©Ù§ [38]3òÌN�.���(JA^��Y�'�A^¥uyù�¯K"

• [105]Ú [106]L²Twitter-LDAÚLDAÚATM�'�§éuÆSÌKc®kXéÐ��J"

• dTwitter-LDA���3�YA^¥�`:Ò´�±é�B/£�Tweet�¡þ�ÚO¯K§X��8Ü¥kõ�^�£�'�¯K§/{IoÚ0Ú/cnê03õ�^�£�'�Tweet¥�y" LDAÚATMéuùa�ÚO¯K3Gibbsæ��?1����VÇO�"

Øùn��.§ [74]ò�c�Labeled-LDA [75]��A^�Tweetsþ§§rz�HashTagÑ����I\"ù«�{���Ð?´�±|^^r3¦^HashTag�/¤�Ûª�ª§�´ù��.���":´ØU��A^�¤kØ�¹HashTag�Tweet¥�"

3.3.2 ÌÌÌKKK���...333Twitter¥¥¥���AAA^]]]ÔÔÔ

ÌK�.3Twitter¥�A^]ÔÌ��±©�ü�a"1�a´du8cÌN�.ïÄ�Û�5�5�§~XÌKê8�(½ÚÌK�.�µd"ùüÜ©3c¡Ñk�[�?ا3d·�ÑL"1�a´duTwitterêâ���5�]Ôµ

• êâ���5�±99��#�Ý"���8c��§�vkó���)ûÌK�.��5�Üݱ93�ÆS"

• DÑ�"Tweets Ñ´é�¿Ö��§¤±�Oi£c¤!#)c!�ä^�!ÎÒ�ó��'å±ÊÏ�ä©�êâké��O\§ù�¯K��éÐ)û"d§�~���µc®§X§/love0§”tomorrow”§du4p�ÑyªÇ�éu©��÷?ÖE¤�½�K�"Ïd3ÌK�.���µ�.´�«ü$ùc®�K��{"

• êâ�õA�'é5"Twitter��þ´�«�¬'X�ä§�c�¦^�ÌK�. �Àù�¡��Ķ¿�=¦üü3SNþ§Tweets���äkéõA:§XØÓa.�©a£�M!=1!£E��¤§¦^Hashtag§�k�mA���"¤±`Twitter��8Üõ�K�A�"§XÛÓ�òùA�K\ÌK�.?1ï�E,´��m��)û�¯K"

8Äu^r�Tweetàܧ,�ÄuàÜ��©�¦^LDA§�±n)�LDA½öATM§�´�BuQãÜ6þÚ'�þ��ߧ·�òù«�{��ATM Ø´LDA"

13

Page 15: ÌÌÌKKK˝˝˝333'''˛˛˛ ÷÷÷¥¥¥˙˙˙AAA^^^net.pku.edu.cn/~zhaoxin/Topic-model-xin-zhao-wayne.pdfComputer Networks and Distributed Systems Lab Peking University Beijing, China

Figure 18: Ä�ÌK�.«¿ã [8]"

Figure 19: �ê��ÅëY�mÌK�.«¿ã [95]"

3.4 ���SSS©©©���666���÷÷÷

DÚ�ÌK�.X [11, 37]Ñ´r©�8Üw�����·��8Ü5?n"�Xpé��ØäuЧ©�Ñ´Ä��5�§�é{`·�¡é�´©�6"3©�6¥§z�©�Ø©�SNA�§���~'����Ò´�mÏ�"XÛ3ÌK�.¥�x�mÏ�!XÛlÌK�.��Ý5�Ä�S©�6��÷ó�§ùÑ´�~'��¯K§3d§·�{�/o(�eµ1¤ÌK�.¥XÛÚ\�mÏf¶2¤ÌN�.3�S©�6þ�~�?Ö"

3.4.1 ÌÌÌKKK���...¥¥¥ÚÚÚ\\\���mmmÏÏÏfff

David M. BleiÄkJÑÄ��ÌK�. [8]§·�3ã 18�ÑÄ�ÌK�.���«¿ã"éuù�Ä�ÌK�.§Ä�g�´ëêzαÚβ§Ø´b½���m���¤k�m:Ñ´�Ó�§/ªz/`§·�¦^αtÚβt ,�|^ê��Åb�§=αt|αt−1 ∼ N (αt−1, δ

2I)Úβt|βt−1 ∼ N (βt−1, δ2I)"Ï

�/`§�c�m:�&EÑ�6uc���m:�&E" [2]æ^Ó��?nÃã§�´¦^���½�Ý�wÄ�mI�"��þù�{Ñ´ò�mlÑz§�Ò´Ï�þ`�“1?n”��{"�´éu�A^§�Uéu�m�âÝ�¦�p½öéJUì1y©êâ" [95]JÑ���ê

��ÅëY�m�.§Xã 19¤«"Ø��c�ó�¦^ê��Åb�§ [95]@�éu��©�Ø©�&E��±§�mI\�´��&E",�ÏLÌK©Ù&E5Ó�'éå5c®Ú�mI\"5¿§éu [95]E,b½ÌK8ÜØ�X�mCz Cz" [92]?�Útµù«b�§JÑ,�«ëY�mÌN�.§3ù��.¥§ÌK8Ü�X�mCz Cz"

3.4.2 ���SSS©©©���666���~~~������©©©������÷÷÷???ÖÖÖ

3d·�{�/0��ÄuÌN�.��S©�6�÷?Ö"

• Burst�÷"�ÿBurstÄk�KleinbergJÑ [42]§��´©�6¥�c½öá�Ñy�cªþ�â,O\§ ù«cªþ�wÍO\ék�UéAX,­��¯�§¤±Äuburst��{²~�

14

Page 16: ÌÌÌKKK˝˝˝333'''˛˛˛ ÷÷÷¥¥¥˙˙˙AAA^^^net.pku.edu.cn/~zhaoxin/Topic-model-xin-zhao-wayne.pdfComputer Networks and Distributed Systems Lab Peking University Beijing, China

Figure 20: ©�6¥�ÌK�S?z«¿ã [59]"

^u¯�uÿ [28]" 3ùp§·�!Ø�´“Bursty Topic”§¤±Ø´�ü�c®�burst§ ´�,�9:{K"�ÿõ�6�m�burstyÌKÄk3 [94]�Jѧ [94]Äkò�my©¤ã§,�©O¦^pLSA [37]ÆS�Ñz��m¡ãSÜ�ÌK"��ÿburstyÌK§ [94]JÑü��{µ1¤¦^c��mãSÜ�ÌK?1²w¶2¤ØÓ6�m�ÌKp\r"�´ [94]¥��{¦^�ÓÚ�©�6§�Ò´`éuÓ��ÌK���§õ�©�6�m��mAT'��C" [93]?�Útµù���§ïÄlõ�ÉÚ�©�6�÷burstyÌK"éu�©�6§~XÆâ©�6§du�gïÄ+��ØÓ�kë�Æö�ØÓ§ék�UéuÓ��ÌK��9kX���É"

• ©�6¥�ÌK�S?z"ùÜ©SN¢Sþ3�c?Ø�¯K“ÌK�.¥XÛÚ\�mÏf”¥®²��Ü©�9"3d§duù�¯K�­�5§·�ò2glA^��Ý5?1?Ø" [59]1�gw«/JÑ?1©�6¥�ÌK�S?z©Û"lù�¯KIK§·��±w�kü�Ì�¯Kµ1¤1��´XÛ�÷ÌK¶2¤1��´XÛ?1ÌK��S©Û" [59]¦^�é��{ü��{§�´Äky©êâã§,�©ãÆS��ÌK8ܧ,��â3ëY�ü��mãS�ÌK�qÝéÙïáó�'X"ã 20�Ñ��¯��ÌK�S?z�~f"·��±uyù«Ðy¯�SÜÌK�?z��{¢SþJø�«éu©�6�Á��ª"¢Sþ§�y�Timeline9�æ^�Ó��ª§�´z�f��Ñ´��äN¯� Ø´��ÌK"

3.5 ���äää(((���êêêâââ������÷÷÷

�ä(�´Ø�mA��,��©��N\&E§§�3u�«a.�êâ8Ü¥µX©z¥�Ú^'X!Æ�¥�Ðl'X!��¥�ó�'X"ùó��±�Ð/�Ï©Û©���¹Â"¢Sþéu�êâa.5`§~X�¬'X�äêâ§ó�'X��Ò´�«êâa.§?uÚ©�Ó�­��/ "3d§·�Á0��e�~��ò�ä(�&EK\ÌK�.��{"

|||^���qqq555

·�Äk0��eRelational topic model(RTM) [23]§ã21�Ñ��{ü)¤L§«¿ã"·��±w��IO�LDA [11]�'§RTMõ1���Ú½§ù��c�?Ø�link-plsa-lda [64]�Ä�g��~aq§Ñ´ÁãlÌK©Ù��q55�Ä?�Ú)¤N\�ó�(�"¢Sþ§ù´¦^��Û¹�b�µXJü�©��mkXó�'X§@o¦��m�ÌK©ÙAT���q"@o�½©��ÌK©Ù��§XÛ�Äó�Qº [23]�Äü�ªf��ó�VǼê:

¼ê1µφδ(y = 1) = δ(ηT (zd · zd′) + ν), (2)¼ê2µφe(y = 1) = exp(ηT (zd · zd′) + ν), (3)

9http://newstimeline.googlelabs.com

15

Page 17: ÌÌÌKKK˝˝˝333'''˛˛˛ ÷÷÷¥¥¥˙˙˙AAA^^^net.pku.edu.cn/~zhaoxin/Topic-model-xin-zhao-wayne.pdfComputer Networks and Distributed Systems Lab Peking University Beijing, China

Figure 21: Relational topic model�)¤L§ [23]"

Ù¥1��¼ê´~^�Sigmoid¼ê§1��´~^��ê¼ê"[48] ÿÐ [23]§ÏLÚ\�¬'X3���ó�'X/¤¼ê¥"�Ò´`§ü�(:/¤

ó�'X´duü�Ï�µ1¤1��´ÌK©Ù��q5¶2¤,��Ï�´�«lá'X��q5" [97]�Ó��Ä©�Úó��)¤§Ú[23]ØÓ�´§ý­�Äzgó�¯��/¤L§µÄk§���ó�¯�ÀJ��ÌKI\¶,�ÄuTÌK�ë��T¯��ü�(:?1+|£½ö�«¤I\�æ�¶��Äuùü�+|I\5û½��ó�¯�/¤�VÇ"

|||^555KKKzzz������{{{

5Kz£Regularization¤´�«3�`z!ÚO!ÅìÆS¥²~¦^��{10"ÙÄ�g�´éu�.��`z¼êþV\���§ÏLù��¦��.;�LÝ[Ü�¾�ÆS¯K"¢Sþ§éu?Û�õêÌK�.§~XIO�LDAÚpLSA§Ù¦)L§Ñ´���`z�L§§Ù8I¼êÒ´¦�T�.3��8Üþ�q,��" [56, 20]JѦ^�ä5KzÏf3ÌK�."Ä�g��´XJü�(:�3ó�'X§@o§��m�½�3X�Ýþþ��q5"�äN�~f§XJü��öQ²Ü�uLLØ©§@o¦��m�ÌK,�©ÙAT�q¶XJü��C�/�ÑkXéu��¯����§@o¦�¤���ÌKAT�q"

ÚÚÚ\\\���«««CCCþþþ???111ÛÛÛªªªàààaaa

þ¡¤`��{Ñ´w«/|^�ä(�A�?1àa/¤�äf(��{",�«�{´Øwª/éuó�?1ï�§¢SþAuthor Topic Model [77]�±w��«Ûªàa��{§�Ò´`·��±UìÌK�©Ùéu�ö?1àa§~Xò�ö©�¦äk��ê��ÌKS�+N" [110] ÄuAuthor-Topic Model [77]JÑü�ù���.§3ÌK�.¥O\��“f(�”Cþ£~X§3�¬'X�䥧f(���«¤§¤kf(:ÑÚù“f(�”ïáéX§/¤�«aq(/(�§ù�cü«�{kX���ØÓ"ÙÄ�g�´3Author-Topic Model�Ä:þÚ\“�«”Cþ§,��½z��«3�ö8Üþk��õ�ª©Ù"

3.6 ���ÙÙÙ¦¦¦AAA^

3þãÙ!§·��[/0��ÌK�.�A^ÚC/"3ù�Ù!§·�{�/£�eÌK�.��Ù¦A^"3ùp§·�Ø2�\[!§ ´æ^�«÷*��Ý5�V`�eùA^"

ggg,,,���óóó???nnn

IO��ó�.§XLDAÚpLSA§éJ��A^��DÚ�g,�ó?n?Ö¥§Xc5I5!é{©Û��"<� ÿÐùÄ��ÌK�.§æ^�?��d�.5;�?nù?Ö"Xc5I5 [88]!cÂ�Ü [15, 21, 12]!é(�){©Û [13, 41, 31]!á�£O [96, 106]!·¶¢N©Û [99, 67]!V��÷ [14, 39, 61, 68, 104, 103]!Á�)¤ [82, 26, 22]��"3ùA^¥§ÌK�.�A^ �z���d�.�A^"

10http://en.wikipedia.org/wiki/Regularization (mathematics)

16

Page 18: ÌÌÌKKK˝˝˝333'''˛˛˛ ÷÷÷¥¥¥˙˙˙AAA^^^net.pku.edu.cn/~zhaoxin/Topic-model-xin-zhao-wayne.pdfComputer Networks and Distributed Systems Lab Peking University Beijing, China

&&&EEEuuu¢¢¢

Ù¢�cùL��Ù!§kéõ?ÖÑ´Ú&Eu¢�'�"3d§·�2QãeÙ¦��UA^ÌK�.�&Eu¢?Ö"XÄ��u¢?Ö [98, 108, 45]!;[Ïé [27]!&EÄ� [102, 25]!I5 [108, 58, 80, 16, 50, 91]��"

4 ÌÌÌKKK���...óóóäää

3ùp§·�{���e�Ä��ÌK�.ó䥧=ø���ÆâïĦ^"k [6, 18, 73, 35, 55,63, 70, 76, 81]"Ù¥�~^���´ [70]"��5`§ÄuGibbsæ��{¢y�óä�Ý'�ú§�´¢yå5��{ü"

5 ooo(((

ÌK�.��þ´�«é©��VÇï���{§¿Ø´���¥@�� �ÚpØ��§�Ø´���¥éu¤k�?ÖÑ´�U�"éuïÄó�ö§7L�âgC�¢S?Ö�I�ÚÌK�.�A5?1g�§l �ä´Ø´AT¦^ÌK�.��{"F"�©éu©��÷ïÄó�<kX�½�ë��^"

References[1] Loulwah AlSumait, Daniel BarbarS, James Gentle, and Carlotta Domeniconi. Topic significance ranking

of lda generative models. In ECML, 2009.

[2] Loulwah AlSumait, Daniel Barbara, and Carlotta Domeniconi. On-line lda: Adaptive topic models formining text streams with applications to topic detection and tracking. In Proceedings of the 2008 EighthIEEE International Conference on Data Mining, pages 3–12, Washington, DC, USA, 2008. IEEE ComputerSociety.

[3] Arthur Asuncion, Padhraic Smyth, and Max Welling. Asynchronous distributed learning of topic models.In NIPS, pages 81–88, 2008.

[4] Arindam Banerjee and Sugato Basu. Topic Models over Text Streams: A Study of Batch and OnlineUnsupervised Learning. In SDM. SIAM, 2007.

[5] Christopher M. Bishop. Pattern Recognition and Machine Learning (Information Science and Statistics).Springer-Verlag New York, Inc., Secaucus, NJ, USA, 2006.

[6] David M. Blei. lda-c, 2003.

[7] David M. Blei, Thomas Griffiths, Michael Jordan, and Joshua Tenenbaum. Hierarchical topic models andthe nested chinese restaurant process. In NIPS, 2003.

[8] David M. Blei and John D. Lafferty. Dynamic topic models. In ICML, 2006.

[9] David M. Blei and John D. Lafferty. A correlated topic model of science. AAS, 1(1):17–35, 2007.

[10] David M. Blei and Jon D. McAuliffe. Supervised topic models. In NIPS, 2007.

[11] David M. Blei, Andrew Y. Ng, and Michael I. Jordan. Latent dirichlet allocation. J. Mach. Learn. Res.,3:993–1022, March 2003.

[12] Jordan Boyd-Graber and David M. Blei. Putop: Turning predominant senses into a topic model for wsd. InSEMEVAL, 2007.

[13] Jordan Boyd-Graber and David M. Blei. Syntactic topic models. In NIPS, 2008.

[14] Jordan Boyd-Graber and David M. Blei. Multilingual topic models for unaligned text. In UAI, 2009.

17

Page 19: ÌÌÌKKK˝˝˝333'''˛˛˛ ÷÷÷¥¥¥˙˙˙AAA^^^net.pku.edu.cn/~zhaoxin/Topic-model-xin-zhao-wayne.pdfComputer Networks and Distributed Systems Lab Peking University Beijing, China

[15] Jordan Boyd-Graber, David M. Blei, and Xiaojin Zhu. A topic model for word sense disambiguation. InEMNLP, 2007.

[16] S.R.K. Branavan, Harr Chen, Jacob Eisenstein, and Regina Barzilay. Learning document-level semanticproperties from free-text annotations. In Proceedings of ACL-08: HLT, pages 263–271, Columbus, Ohio,June 2008. Association for Computational Linguistics.

[17] Samuel Brody and Noemie Elhadad. An unsupervised aspect-sentiment model for online reviews. InHuman Language Technologies: The 2010 Annual Conference of the North American Chapter of the Asso-ciation for Computational Linguistics, HLT ’10, pages 804–812, Stroudsburg, PA, USA, 2010. Associationfor Computational Linguistics.

[18] Wray L. Buntine. Discrete component analysis, 2009.

[19] Wray L. Buntine. Estimating likelihoods for topic models. In Asian Conference on Machine Learning,2009.

[20] Deng Cai, Qiaozhu Mei, Jiawei Han, and Chengxiang Zhai. Modeling hidden topics on document manifold.In Proceeding of the 17th ACM conference on Information and knowledge management, CIKM ’08, pages911–920, New York, NY, USA, 2008. ACM.

[21] Jun Fu Cai, Wee Sun Lee, and Yee Whye Teh. Nus-ml: Improving word sense disambiguation using topicfeatures. In SEMEVAL, 2007.

[22] Asli Celikyilmaz and Dilek Hakkani-Tur. A hybrid hierarchical model for multi-document summarization.In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, ACL ’10,pages 815–824, Stroudsburg, PA, USA, 2010. Association for Computational Linguistics.

[23] Jonathan Chang and David Blei. Relational topic models for document networks. In AIStats, 2009.

[24] Jonathan Chang, Jordan Boyd-Graber, Chong Wang, Sean Gerrish, and David M. Blei. Reading tea leaves:How humans interpret topic models. In NIPS, 2009.

[25] Chaitanya Chemudugunta, Padhraic Smyth, and Mark Steyvers. Combining concept hierarchies and statis-tical topic models. In Proceeding of the 17th ACM conference on Information and knowledge management,CIKM ’08, pages 1469–1470, New York, NY, USA, 2008. ACM.

[26] Hal Daume, III and Daniel Marcu. Bayesian query-focused summarization. In Proceedings of the 21stInternational Conference on Computational Linguistics and the 44th annual meeting of the Associationfor Computational Linguistics, ACL-44, pages 305–312, Stroudsburg, PA, USA, 2006. Association forComputational Linguistics.

[27] Hui Fang and ChengXiang Zhai. Probabilistic models for expert finding. In Proceedings of the 29thEuropean conference on IR research, ECIR’07, pages 418–430, Berlin, Heidelberg, 2007. Springer-Verlag.

[28] Gabriel Pui Cheong Fung, Jeffrey Xu Yu, Philip S. Yu, and Hongjun Lu. Parameter free bursty eventsdetection in text streams. In Proceedings of the 31st international conference on Very large data bases,2005.

[29] Mark Girolami and Ata Kaban. On an equivalence between plsi and lda. In Proceedings of the 26th annualinternational ACM SIGIR conference on Research and development in informaion retrieval, SIGIR ’03,pages 433–434, New York, NY, USA, 2003. ACM.

[30] T. L. Griffiths and M. Steyvers. Finding scientific topics. Proceedings of the National Academy of Sciences,101(Suppl. 1):5228–5235, April 2004.

[31] Thomas L. Griffiths, Mark Steyvers, David M. Blei, and Joshua B. Tenenbaum. Integrating topics andsyntax. In NIPS, pages 537–544. 2004.

[32] David Hall, Daniel Jurafsky, and Christopher D. Manning. Studying the history of ideas using topic models.In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP ’08,pages 363–371, Stroudsburg, PA, USA, 2008. Association for Computational Linguistics.

18

Page 20: ÌÌÌKKK˝˝˝333'''˛˛˛ ÷÷÷¥¥¥˙˙˙AAA^^^net.pku.edu.cn/~zhaoxin/Topic-model-xin-zhao-wayne.pdfComputer Networks and Distributed Systems Lab Peking University Beijing, China

[33] Qi He, Bi Chen, Jian Pei, Baojun Qiu, Prasenjit Mitra, and Lee Giles. Detecting topic evolution in sci-entific literature: how can citations help? In Proceeding of the 18th ACM conference on Information andknowledge management, CIKM ’09, pages 957–966, New York, NY, USA, 2009. ACM.

[34] Gregor Heinrich. Parameter estimation for text analysis. Technical report, 2004.

[35] Gregor Heinrich. Infinite lda, 2011.

[36] Matthew Hoffman, David M. Blei, and Francis Bach. Online learning for latent dirichlet allocation. InNIPS, 2010.

[37] Thomas Hofmann. Unsupervised learning by probabilistic latent semantic analysis. Mach. Learn., 42:177–196, January 2001.

[38] Liangjie Hong and Brian D. Davison. Empirical study of topic modeling in twitter. In Proceedings of theFirst Workshop on Social Media Analytics, SOMA ’10, pages 80–88, New York, NY, USA, 2010. ACM.

[39] Jagadeesh Jagarlamudi and Hal DaumWIII. Extracting multilingual topics from unaligned comparable cor-pora. pages 444–456, 2010.

[40] Yohan Jo and Alice H. Oh. Aspect and sentiment unification model for online review analysis. In Pro-ceedings of the fourth ACM international conference on Web search and data mining, WSDM ’11, pages815–824, New York, NY, USA, 2011. ACM.

[41] Mark Johnson. Pcfgs, topic models, adaptor grammars, and learning topical collocations and the structureof proper names. 2010.

[42] J. Kleinberg. Bursty and hierarchical structure in streams. Data Mining and Knowledge Discovery, 2003.

[43] Thomas K. Landauer, Peter W. Foltz, and Darrell Laham. An Introduction to Latent Semantic Analysis.Discourse Processes, (25):259–284, 1998.

[44] Peng Li, Jing Jiang, and Yinglin Wang. Generating templates of entity summaries with an entity-aspectmodel and pattern mining. In Proceedings of the 48th Annual Meeting of the Association for ComputationalLinguistics, pages 640–649, Uppsala, Sweden, July 2010. Association for Computational Linguistics.

[45] R.M. Li, R. Kaptein, D. Hiemstra, and J. Kamps. Exploring topic-based language models for effectiveweb information retrieval. In E. Hoenkamp, M. de Cock, and V. Hoste, editors, Proceedings of the Dutch-Belgian Information Retrieval Workshop (DIR 2008), pages 65–71, Enschede, April 2008. Neslia Panicu-lata.

[46] Wei Li and Andrew McCallum. Pachinko allocation: Dag-structured mixture models of topic correlations.In ICML, 2006.

[47] Chenghua Lin and Yulan He. Joint sentiment/topic model for sentiment analysis. In Proceeding of the 18thACM conference on Information and knowledge management, CIKM ’09, pages 375–384, New York, NY,USA, 2009. ACM.

[48] Yan Liu, Alexandru Niculescu-Mizil, and Wojciech Gryc. Topic-link lda: joint models of topic and authorcommunity. In Proceedings of the 26th Annual International Conference on Machine Learning, ICML ’09,pages 665–672, New York, NY, USA, 2009. ACM.

[49] Zhiyuan Liu, Yuzhou Zhang, Edward Y. Chang, and Maosong Sun. Plda+: Parallel latent dirichlet allocationwith data placement and pipeline processing. ACM Trans. Intell. Syst. Technol., 2:26:1–26:18, May 2011.

[50] Caimei Lu, Xiaohua Hu, Xin Chen, Jung-Ran Park, TingTing He, and Zhoujun Li. The topic-perspectivemodel for social tagging systems. In Proceedings of the 16th ACM SIGKDD international conference onKnowledge discovery and data mining, KDD ’10, pages 683–692, New York, NY, USA, 2010. ACM.

[51] Yue Lu and Chengxiang Zhai. Opinion integration through semi-supervised topic modeling. In Proceedingof the 17th international conference on World Wide Web, WWW ’08, pages 121–130, New York, NY, USA,2008. ACM.

19

Page 21: ÌÌÌKKK˝˝˝333'''˛˛˛ ÷÷÷¥¥¥˙˙˙AAA^^^net.pku.edu.cn/~zhaoxin/Topic-model-xin-zhao-wayne.pdfComputer Networks and Distributed Systems Lab Peking University Beijing, China

[52] Gideon S. Mann, David Mimno, and Andrew McCallum. Bibliometric impact measures leveraging topicanalysis. In Proceedings of the 6th ACM/IEEE-CS joint conference on Digital libraries, JCDL ’06, pages65–74, New York, NY, USA, 2006. ACM.

[53] Andrew McCallum. Multi-label text classification with a mixture model trained by em. In AAAI’99 Work-shop on Text Learning, 1999.

[54] Andrew McCallum, Xuerui Wang, and Andres Corrada-Emmanuel. Topic and role discovery in socialnetworks with experiments on enron and academic email. J. Artif. Int. Res., 30:249–272, October 2007.

[55] Andrew Kachites McCallum. Mallet: A machine learning for language toolkit, 2002.

[56] Qiaozhu Mei, Deng Cai, Duo Zhang, and ChengXiang Zhai. Topic modeling with network regularization.In Proceeding of the 17th international conference on World Wide Web, WWW ’08, pages 101–110, NewYork, NY, USA, 2008. ACM.

[57] Qiaozhu Mei, Xu Ling, Matthew Wondra, Hang Su, and ChengXiang Zhai. Topic sentiment mixture:modeling facets and opinions in weblogs. In Proceedings of the 16th international conference on WorldWide Web, WWW ’07, pages 171–180, New York, NY, USA, 2007. ACM.

[58] Qiaozhu Mei, Xuehua Shen, and ChengXiang Zhai. Automatic labeling of multinomial topic models. InKDD, pages 490–499, 2007.

[59] Qiaozhu Mei and ChengXiang Zhai. Discovering evolutionary theme patterns from text: an exploration oftemporal text mining. In Proceedings of the eleventh ACM SIGKDD international conference on Knowledgediscovery in data mining, KDD ’05, pages 198–207, New York, NY, USA, 2005. ACM.

[60] David Mimno and Andrew McCallum. Topic models conditioned on arbitrary features with dirichlet-multinomial regression. In UAI, 2008.

[61] David Mimno, Hanna Wallach, Jason Naradowsky, David A. Smith, and Andrew McCallum. Polylingualtopic models. In EMNLP, 2009.

[62] Thomas P. Minka. Expectation propagation for approximate bayesian inference. In Proceedings of the 17thConference in Uncertainty in Artificial Intelligence, UAI ’01, pages 362–369, San Francisco, CA, USA,2001. Morgan Kaufmann Publishers Inc.

[63] Ramesh Nallapati. multithreaded lda-c, 2010.

[64] Ramesh M. Nallapati, Amr Ahmed, Eric P. Xing, and William W. Cohen. Joint latent topic models for textand citations. In Proceeding of the 14th ACM SIGKDD international conference on Knowledge discoveryand data mining, KDD ’08, pages 542–550, New York, NY, USA, 2008. ACM.

[65] D. Newman, A. Asuncion, P. Smyth, and M. Welling. Distributed Inference for Latent Dirichlet Allocation.2007.

[66] David Newman, Chaitanya Chemudugunta, and Padhraic Smyth. Statistical entity-topic models. In Pro-ceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining,KDD ’06, pages 680–686, New York, NY, USA, 2006. ACM.

[67] David Newman, Chaitanya Chemudugunta, and Padhraic Smyth. Statistical entity-topic models. In KDD,2006.

[68] Xiaochuan Ni, Jian-Tao Sun, Jian Hu, and Zheng Chen. Mining multilingual topics from wikipedia. InWWW, 2009.

[69] Bo Pang and Lillian Lee. Opinion mining and sentiment analysis. Found. Trends Inf. Retr., 2:1–135, January2008.

[70] Xuan-Hieu Phan and Cam-Tu Nguyen. Gibbslda++, 2007.

20

Page 22: ÌÌÌKKK˝˝˝333'''˛˛˛ ÷÷÷¥¥¥˙˙˙AAA^^^net.pku.edu.cn/~zhaoxin/Topic-model-xin-zhao-wayne.pdfComputer Networks and Distributed Systems Lab Peking University Beijing, China

[71] Jay M. Ponte and W. Bruce Croft. A language modeling approach to information retrieval. In Proceed-ings of the 21st annual international ACM SIGIR conference on Research and development in informationretrieval, SIGIR ’98, pages 275–281, New York, NY, USA, 1998. ACM.

[72] Ian Porteous, David Newman, Alexander Ihler, Arthur Asuncion, Padhraic Smyth, and Max Welling. Fastcollapsed gibbs sampling for latent dirichlet allocation. In Proceeding of the 14th ACM SIGKDD inter-national conference on Knowledge discovery and data mining, KDD ’08, pages 569–577, New York, NY,USA, 2008. ACM.

[73] Radim. gensim, 2009.

[74] Daniel Ramage, Susan Dumais, and Dan Liebling. Characterizing microblogs with topic models. InICWSM, 2010.

[75] Daniel Ramage, David Hall, Ramesh Nallapati, and Christopher D. Manning. Labeled lda: A supervisedtopic model for credit attribution in multi-labeled corpora. In EMNLP, 2009.

[76] Daniel Ramage and Evan Rosen. Stanford topic modeling toolbox, 2009.

[77] Michal Rosen-Zvi, Tom Griffiths, Mark Steyvers, and Padhraic Smyth. The author-topic model for authorsand documents. In UAI, 2004.

[78] G. Salton, A. Wong, and C. S. Yang. A vector space model for automatic indexing. Commun. ACM,18:613–620, November 1975.

[79] Issei Sato and Hiroshi Nakagawa. Topic models with power-law using pitman-yor process. In Proceedingsof the 16th ACM SIGKDD international conference on Knowledge discovery and data mining, KDD ’10,pages 673–682, New York, NY, USA, 2010. ACM.

[80] Yuanlong Shao, Yuan Zhou, Xiaofei He, Deng Cai, and Hujun Bao. Semi-supervised topic modeling forimage annotation. In Proceedings of the 17th ACM international conference on Multimedia, MM ’09, pages521–524, New York, NY, USA, 2009. ACM.

[81] Mark Steyvers and Tom Griffiths. Matlab topic modeling toolbox, 2005.

[82] Jie Tang, Limin Yao, and Dewei Chen. Multi-topic based query-oriented summarization. In SDM, pages1147–1158, 2009.

[83] Y. W. Teh. Dirichlet processes. In Encyclopedia of Machine Learning. Springer, 2010.

[84] Y. W. Teh, M. I. Jordan, M. J. Beal, and D. M. Blei. Hierarchical Dirichlet processes. Journal of theAmerican Statistical Association, 101(476):1566–1581, 2006.

[85] Yee Whye Teh. A hierarchical bayesian language model based on pitman-yor processes. In Proceedings ofthe 21st International Conference on Computational Linguistics and the 44th annual meeting of the Asso-ciation for Computational Linguistics, ACL-44, pages 985–992, Stroudsburg, PA, USA, 2006. Associationfor Computational Linguistics.

[86] Ivan Titov and Ryan McDonald. A joint model of text and aspect ratings for sentiment summarization. InProceedings of ACL-08: HLT, pages 308–316, Columbus, Ohio, June 2008. Association for ComputationalLinguistics.

[87] Ivan Titov and Ryan McDonald. Modeling online reviews with multi-grain topic models. In Proceeding ofthe 17th international conference on World Wide Web, WWW ’08, pages 111–120, New York, NY, USA,2008. ACM.

[88] Kristina Toutanova and Mark Johnson. A bayesian lda-based model for semi-supervised part-of-speech tag-ging. In J.C. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information ProcessingSystems 20, pages 1521–1528. MIT Press, Cambridge, MA, 2008.

[89] Hanna Wallach, David Mimno, and Andrew McCallum. Rethinking lda: Why priors matter. In Y. Bengio,D. Schuurmans, J. Lafferty, C. K. I. Williams, and A. Culotta, editors, Advances in Neural InformationProcessing Systems 22, pages 1973–1981. 2009.

21

Page 23: ÌÌÌKKK˝˝˝333'''˛˛˛ ÷÷÷¥¥¥˙˙˙AAA^^^net.pku.edu.cn/~zhaoxin/Topic-model-xin-zhao-wayne.pdfComputer Networks and Distributed Systems Lab Peking University Beijing, China

[90] Hanna Wallach, Iain Murray, Ruslan Salakhutdinov, and David Mimno. Evaluation methods for topicmodels. In ICML, 2009.

[91] Chong Wang, David Blei, and Fei-Fei Li. Simultaneous image classification and annotation. In CVPR,2009.

[92] Chong Wang, David M. Blei, and David Heckerman. Continuous time dynamic topic models. In UAI,2008.

[93] Xiang Wang, Kai Zhang, Xiaoming Jin, and Dou Shen. Mining common topics from multiple asynchronoustext streams. In Proceedings of the Second ACM International Conference on Web Search and Data Mining,WSDM ’09, pages 192–201, New York, NY, USA, 2009. ACM.

[94] Xuanhui Wang, ChengXiang Zhai, Xiao Hu, and Richard Sproat. Mining correlated bursty topic patternsfrom coordinated text streams. In Proceedings of the 13th ACM SIGKDD international conference onKnowledge discovery and data mining, 2007.

[95] Xuerui Wang and Andrew McCallum. Topics over time: a non-markov continuous-time model of topicaltrends. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery anddata mining, KDD ’06, pages 424–433, New York, NY, USA, 2006. ACM.

[96] Xuerui Wang, Andrew McCallum, and Xing Wei. Topical n-grams: Phrase and topic discovery, with anapplication to information retrieval. In Proceedings of the 2007 Seventh IEEE International Conference onData Mining, pages 697–702, Washington, DC, USA, 2007. IEEE Computer Society.

[97] Xuerui Wang, Natasha Mohanty, and Andrew McCallum. Group and topic discovery from relations andtext. In Proceedings of the 3rd international workshop on Link discovery, LinkKDD ’05, pages 28–35,New York, NY, USA, 2005. ACM.

[98] Xing Wei and W. Bruce Croft. Lda-based document models for ad-hoc retrieval. In Proceedings of the 29thannual international ACM SIGIR conference on Research and development in information retrieval, SIGIR’06, pages 178–185, New York, NY, USA, 2006. ACM.

[99] Gu Xu, Shuang-Hong Yang, and Hang Li. Named entity mining from click-through data using weaklysupervised latent dirichlet allocation. In Proceedings of the 15th ACM SIGKDD international conferenceon Knowledge discovery and data mining, KDD ’09, pages 1365–1374, New York, NY, USA, 2009. ACM.

[100] Feng Yan, Ningyi Xu, and Yuan Qi. Parallel inference for latent dirichlet allocation on graphics processingunits. In NIPS, 2009.

[101] Limin Yao, David Mimno, and Andrew McCallum. Efficient methods for topic model inference on stream-ing document collections. In Proceedings of the 15th ACM SIGKDD international conference on Knowl-edge discovery and data mining, KDD ’09, pages 937–946, New York, NY, USA, 2009. ACM.

[102] Huibin Zhang, Mingjie Zhu, Shuming Shi, and Ji-Rong Wen. Employing topic models for pattern-basedsemantic class discovery. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL andthe 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 1 - Volume1, ACL ’09, pages 459–467, Stroudsburg, PA, USA, 2009. Association for Computational Linguistics.

[103] Bin Zhao and Eric P. Xing. Hm-bitam: Bilingual topic exploration, word alignment, and translation. InNIPS, 2007.

[104] Bing Zhao and Eric P. Xing. Bitam: Bilingual topic admixture models for word alignment. In ACL, 2006.

[105] Wayne Xin Zhao, Jing Jiang, Jianshu Weng, Jing He, Ee-Peng Lim, Hongfei Yan, and Xiaoming Li. Com-paring twitter and traditional media using topic models. In ECIR, pages 338–349, 2011.

[106] Xin Zhao, Jing Jiang, Jing He, Yang Song, Palakorn Achanauparp, Ee-Peng Lim, and Xiaoming Li. Topicalkeyphrase extraction from twitter. In ACL-HLT, 2011.

22

Page 24: ÌÌÌKKK˝˝˝333'''˛˛˛ ÷÷÷¥¥¥˙˙˙AAA^^^net.pku.edu.cn/~zhaoxin/Topic-model-xin-zhao-wayne.pdfComputer Networks and Distributed Systems Lab Peking University Beijing, China

[107] Xin Zhao, Jing Jiang, Hongfei Yan, and Xiaoming Li. Jointly modeling aspects and opinions with aMaxEnt-LDA hybrid. In Proceedings of the 2010 Conference on Empirical Methods in Natural LanguageProcessing, pages 56–65, Cambridge, MA, October 2010. Association for Computational Linguistics.

[108] Ding Zhou, Jiang Bian, Shuyi Zheng, Giles Lee, and Hongyuan Zha. Exploring social annotations for in-formation retrieval. In Proceedings of the 17th International World Wide Web Conference, Beijing, Peking,2008.

[109] Ding Zhou, Xiang Ji, Hongyuan Zha, and C. Lee Giles. Topic evolution and social interactions: how authorseffect research. In Proceedings of the 15th ACM international conference on Information and knowledgemanagement, CIKM ’06, pages 248–257, New York, NY, USA, 2006. ACM.

[110] Ding Zhou, Eren Manavoglu, Jia Li, C. Lee Giles, and Hongyuan Zha. Probabilistic models for discoveringe-communities. In Proceedings of the 15th international conference on World Wide Web, WWW ’06, pages173–182, New York, NY, USA, 2006. ACM.

23

Page 25: ÌÌÌKKK˝˝˝333'''˛˛˛ ÷÷÷¥¥¥˙˙˙AAA^^^net.pku.edu.cn/~zhaoxin/Topic-model-xin-zhao-wayne.pdfComputer Networks and Distributed Systems Lab Peking University Beijing, China

AppendixTÜ©�ÑLDA����ëêí�L§§��ß��§·�ò�ܦ^=�5?1£ãÚí�"

Let α = (α1, .., αK) be the hyper-parameters for document-topic distribution(i.e. θd = {p(z|d)}z). Let β =(β1, .., βV ) be the hyper-parameters for topic-word distribution(i.e. φz = {p(w|z)}w). The prior probabilities onparameters are defined in a Dirichlet distribution

p(θ|α) =D∏

m=1

(Γ(∑K

k=1 αk)∏Kk=1 Γ(αk)

K∏k=1

θαk−1m,k

), (4)

p(φ|β) =K∏

k=1

(Γ(∑V

v=1 βv)∏Vv=1 Γ(βv)

V∏v=1

φβv−1k,v

).

The probabilities of all hidden variables given parameters {θd}d is defined in a multinomial distribution

p(z|θ) =D∏

m=1

K∏k=1

θnm,k

m,k , (5)

where nm,k denotes the count of words assigned to topic k in document m.The probabilities of all observed variables(words) given parameters {φz}z and all the hidden variables is

defined in a multinomial distribution

p(w|z, φ) =K∏

k=1

V∏v=1

φnk,v

k,v , (6)

where nk,v denotes the count of word v assigned to topic k in corpus.Noting that, in these two formulas, we assume that given parameters all the variables are indepedent of hyper-

parameters.Given the hyper-parameters, the joint probability of all the variables (including parameters) is defined

p(w, z, θ, φ|α, β) = p(w|z, θ)p(z|φ)p(φ|β)p(θ|α). (7)

So after deriving the joint probability, we integrate all the parameters

p(w, z|α, β) =∫

θ

∫φ

p(w|z, θ)p(z|φ)p(φ|β)p(θ|α)dφdθ, (8)

=∫

θ

∫φ

(D∏

m=1

K∏k=1

θnm,k

m,k

)(K∏

k=1

V∏v=1

φnk,v

k,v

)(

K∏k=1

(Γ(∑V

v=1 βv)∏Vv=1 Γ(βv)

V∏v=1

φβv−1k,v

))( D∏m=1

(Γ(∑K

k=1 αk)∏Kk=1 Γ(αk)

K∏k=1

θαk−1m,k

))dφdθ,

=∫

θ

((Γ(∑K

k=1 αk)∏Kk=1 Γ(αk)

)D D∏m=1

K∏k=1

θnm,k+αk−1m,k

)dθ

∫φ

((Γ(∑V

v=1 βv)∏Vv=1 Γ(βv)

)K K∏k=1

V∏v=1

φnk,v+φv−1k,v

)dφ

=(Γ(

∑Kk=1 αk)∏K

k=1 Γ(αk)

)D D∏m=1

∫θ

(K∏

k=1

θnm,k+αk−1m,k

)dθ

(Γ(∑V

v=1 βv)∏Vv=1 Γ(βv)

)K K∏k=1

∫φ

(V∏

v=1

φnk,v+φv−1k,v

)dφ

=(Γ(

∑Kk=1 αk)∏K

k=1 Γ(αk)

)D D∏m=1

∏Kk=1 Γ(αk + nm,k)

Γ(∑K

k=1 αk + nm,k)(Γ(∑V

v=1 βv)∏Vv=1 Γ(βv)

)K K∏k=1

∏Vv=1 Γ(βv + nk,v)

Γ(∑V

v=1 βv + nk,v)

24

Page 26: ÌÌÌKKK˝˝˝333'''˛˛˛ ÷÷÷¥¥¥˙˙˙AAA^^^net.pku.edu.cn/~zhaoxin/Topic-model-xin-zhao-wayne.pdfComputer Networks and Distributed Systems Lab Peking University Beijing, China

p(zi = t|w, z¬i, α, β) =p(zi,w, z¬i, α, β)p(w, z¬i, α, β)

(9)

=p(w, z, α, β)

p(w¬i, wi, z¬i, α, β)

=p(w, z, α, β)

p(w¬i, z¬i, α, β)p(wi|w¬i, z¬i, α, β)

∝ p(w, z|α, β)p(w¬i, z¬i|α, β)

=

(Γ(

∑Kk=1 αk)∏K

k=1 Γ(αk)

)D∏Dm=1

∏Kk=1 Γ(αk+nm,k)

Γ(∑K

k=1 αk+nm,k)

(Γ(

∑Vv=1 βv)∏V

v=1 Γ(βv)

)K ∏Kk=1

∏Vv=1 Γ(βv+nk,v)

Γ(∑V

v=1 βv+nk,v)(Γ(

∑Kk=1 αk)∏K

k=1 Γ(αk)

)D∏Dm=1

∏Kk=1 Γ(αk+n−i

m,k)

Γ(∑K

k=1 αk+n−im,k)

(Γ(

∑Vv=1 βv)∏V

v=1 Γ(βv)

)K ∏Kk=1

∏Vv=1 Γ(βv+n−i

k,v)

Γ(∑V

v=1 βv+n−ik,v)

=αt + n−i

m,t∑Kk=1 αk + n−i

m,k

βv + n−it,v∑V

v=1 βv + n−it,v

.

The derivations above used the results in Equation 8 and also one important property of Gamma function Γ(x +1) = xΓ(x).

25