Upload
others
View
9
Download
0
Embed Size (px)
Citation preview
Probabilistic Topic Models -Latent Dirichlet Allocation
Machine Learning Reading Group
March 2014
ML Reading Group Probabilistic Topic Models 1 / 12
Related Papers
D. Blei. Probabilistic topic models. Communications of the ACM,55(4):77-84, 2012.
D. Blei, A. Ng, M. Jordan. Latent Dirichlet allocation. Journal ofMachine Learning Research 3:993-1022, 2003.
ML Reading Group Probabilistic Topic Models 2 / 12
Motivation
Searching and exploring documents based on the themes that runthrough them
Rather than finding documents through keyword search alone, wemight first find the theme that we are interested in, and then examinethe documents related to that theme.
Probabilistic topic modeling: a suite of algorithms that aim todiscover and annotate large archives of documents with thematicinformation.
Note: these algorithms, sometimes in different names, are used forother data types (audio, image, video...)
ML Reading Group Probabilistic Topic Models 3 / 12
Topic Models (1)
Unigram model: the words of every document are drawnindependently from a single multinomial distribution
p(w) =∏
n
p(wn) (1)
ML Reading Group Probabilistic Topic Models 4 / 12
Topic Models (2)
Mixture of unigrams: each document is generated by first choosing atopic z and then generating N words independently from theconditional multinomial p(w |z)
p(w) =∑
z
p(z)∏
n
p(wn|z) (2)
ML Reading Group Probabilistic Topic Models 5 / 12
Topic Models (3)
Probabilistic latent semantic indexing (PLSI): it captures thepossibility that a document d may contain multiple topics
p(d ,wn) = p(d)∑
z
p(wn|z)p(z |d) (3)
ML Reading Group Probabilistic Topic Models 6 / 12
Latent Dirichlet Allocation (1)
Latent Dirichlet Allocation (LDA): expands PLSI by introducing priorson probability distributions
Better generalisability on new documents
ML Reading Group Probabilistic Topic Models 7 / 12
Latent Dirichlet Allocation (2)
Generative Process1 Randomly choose a distribution over topics2 For each word in the document
Randomly choose a topic from the distribution over topics in step 1Randomly choose a word from the corresponding distribution over thevocabulary
ML Reading Group Probabilistic Topic Models 8 / 12
Latent Dirichlet Allocation (3)
Graphical Model
ML Reading Group Probabilistic Topic Models 9 / 12
Latent Dirichlet Allocation (4)
Posterior Computation
Posterior is intractable - need to approximate it
Variational inference
MCMC - Gibbs sampling
ML Reading Group Probabilistic Topic Models 10 / 12
Latent Dirichlet Allocation (5)
Extensions
LDA can be exteded by relaxing some of the original assumptions
Bag-of-words: Not suitable for language generation
Solution: Integrating syntax
Bag-of-documents: Not suitable for chronologically ordereddocuments
Solution: Dynamic topic models
Number of topics: Assumed to be known and fixed
Solution: Bayesian nonparametric topic models
ML Reading Group Probabilistic Topic Models 11 / 12
Latent Dirichlet Allocation (6)
Open Issues
Evaluation and model checking: Interpretability over goodness of fit.
Visualization and user interfaces: Intuitive ways to visualise topics.
Data discovery : Seeking help of domain experts.
ML Reading Group Probabilistic Topic Models 12 / 12