← Back to papers
2014 NIPS NeurIPS 2014

Analysis of Variational Bayesian Latent Dirichlet Allocation: Weaker Sparsity Than MAP

Abstract

Latent Dirichlet allocation (LDA) is a popular generative model of various objects such as texts and images, where an object is expressed as a mixture of latent topics. In this paper, we theoretically investigate variational Bayesian (VB) learning in LDA. More specifically, we analytically derive the leading term of the VB free energy under an asymptotic setup, and show that there exist transition thresholds in Dirichlet hyperparameters around which the sparsity-inducing behavior drastically changes. Then we further theoretically reveal the notable phenomenon that VB tends to induce weaker sparsity than MAP in the LDA model, which is opposed to other models. We experimentally demonstrate the practical validity of our asymptotic theory on real-world Last.FM music data.

🌉 Interdisciplinary Bridge — Artificial Intelligence and Deep Learning and Natural Language Processing
📈 Trend Setter — Text Representation
🧭 Keyword Pioneer — sparsity analysis
🐝 Cross-Pollinator — Artificial Intelligence, Computer Science, Computer Vision, Data Science & Analytics, Deep Learning, Healthcare & Medicine, Interdisciplinary, Knowledge & Reasoning, Machine Learning, Mathematics & Optimization, Natural Language Processing, Reinforcement Learning, Robotics, Security & Privacy, Speech & Audio
🐣 Hot Topic Early Bird — topic modeling

Authors

Shinichi Nakajima , Issei Sato , Masashi Sugiyama , Kazuho Watanabe , Hiroko Kobayashi

Topics

Artificial Intelligence > Bayesian & Probabilistic > Bayesian Learning Machine Learning > Optimization & Theory > Bayesian Inference Deep Learning > Models > Variational Inference Natural Language Processing > Applications > Text Classification Natural Language Processing > Resources & Methods > Text Representation Machine Learning > Bayesian & Probabilistic > Bayesian Learning Machine Learning > Bayesian & Probabilistic > Bayesian Inference Machine Learning > Bayesian & Probabilistic > Variational Inference Natural Language Processing > Applications > Topic Modeling

Keywords

variational inference bayesian learning bayesian inference latent dirichlet allocation topic modeling variational bayesian free energy sparsity analysis topic model dirichlet hyperparameter
Download PDF

Related papers

Information-based learning by agents in unbounded state spaces 2014
Stochastic Gradient Descent, Weighted Sampling, and the Randomized Kaczmarz algorithm 2014
Partition-wise Linear Models 2014
Active Regression by Stratification 2014
Cone-Constrained Principal Component Analysis 2014