← Back to papers
2012 NIPS NeurIPS 2012

A Better Way to Pretrain Deep Boltzmann Machines

Abstract

We describe how the pre-training algorithm for Deep Boltzmann Machines (DBMs) is related to the pre-training algorithm for Deep Belief Networks and we show that under certain conditions, the pre-training procedure improves the variational lower bound of a two-hidden-layer DBM. Based on this analysis, we develop a different method of pre-training DBMs that distributes the modelling work more evenly over the hidden layers. Our results on the MNIST and NORB datasets demonstrate that the new pre-training algorithm allows us to learn better generative models.

📈 Trend Setter — Pretraining
🧭 Keyword Pioneer — pretraining
🐣 Hot Topic Early Bird — generative model
🐝 Cross-Pollinator — Artificial Intelligence, Computer Vision, Data Science & Analytics, Deep Learning, Healthcare & Medicine, Interdisciplinary, Knowledge & Reasoning, Machine Learning, Mathematics & Optimization, Natural Language Processing, Reinforcement Learning, Robotics, Speech & Audio
🌉 Interdisciplinary Bridge — Deep Learning and Machine Learning

Authors

Geoffrey E. Hinton , Ruslan Salakhutdinov

Topics

Deep Learning > Architectures > Neural Networks Deep Learning > Models > Generative Models Deep Learning > Techniques > Pretraining Machine Learning > Core Methods > Probabilistic Modeling Machine Learning > Learning Types > Representation Learning

Keywords

variational inference unsupervised feature learning pretraining hidden layer variational pretraining greedy layerwise training generative model deep boltzmann machine variational lower bound deep belief network pretraining algorithm
Download PDF

Related papers

Kernel Hyperalignment 2012
Fused sparsity and robust estimation for linear models with unknown variance 2012
Slice sampling normalized kernel-weighted completely random measure mixture models 2012
Scaling MPE Inference for Constrained Continuous Markov Random Fields with Consensus Optimization 2012
Matrix reconstruction with the local max norm 2012