← Back to papers
2005 JMLR JMLR 2005

Convergence Theorems for Generalized Alternating Minimization Procedures

Abstract

The EM algorithm is widely used to develop iterative parameter estimation procedures for statistical models. In cases where these procedures strictly follow the EM formulation, the convergence properties of the estimation procedures are well understood. In some instances there are practical reasons to develop procedures that do not strictly fall within the EM framework. We study EM variants in which the E-step is not performed exactly, either to obtain improved rates of convergence, or due to approximations needed to compute statistics under a model family over which E-steps cannot be realized. Since these variants are not EM procedures, the standard (G)EM convergence results do not apply to them. We present an information geometric framework for describing such algorithms and analyzing their convergence properties. We apply this framework to analyze the convergence properties of incremental EM and variational EM. For incremental EM, we discuss conditions under these algorithms converge in likelihood. For variational EM, we show how the E-step approximation prevents convergence to local maxima in likelihood. [abs] [ pdf ][ bib ] © JMLR 2005. (edit, beta)

🌉 Interdisciplinary Bridge — Machine Learning and Mathematics & Optimization
📈 Trend Setter — Stochastic Methods
🧭 Keyword Pioneer — variational em
🐝 Cross-Pollinator — Artificial Intelligence, Computer Vision, Data Science & Analytics, Deep Learning, Healthcare & Medicine, Interdisciplinary, Machine Learning, Mathematics & Optimization, Reinforcement Learning, Robotics, Speech & Audio
🐣 Hot Topic Early Bird — variational inference

Authors

Asela Gunawardana , William Byrne

Topics

Machine Learning > Optimization & Theory > Learning Theory Machine Learning > Optimization & Theory > Optimization Mathematics & Optimization > Optimization > Stochastic Methods Machine Learning > Bayesian & Probabilistic > Variational Inference

Keywords

variational inference em algorithm convergence analysis expectation maximization variational em alternating minimization information geometry incremental em
Download PDF

Related papers

Diffusion Kernels on Statistical Manifolds 2005
Learning with Decision Lists of Data-Dependent Features 2005
Multiclass Classification with Multi-Prototype Support Vector Machines 2005
Loopy Belief Propagation: Convergence and Effects of Message Errors 2005
Efficient Computation of Gapped Substring Kernels on Large Alphabets 2005