← Back to papers
2006 NIPS NeurIPS 2006

Efficient sparse coding algorithms

Abstract

Sparse coding provides a class of algorithms for finding succinct representations of stimuli; given only unlabeled input data, it discovers basis functions that capture higher-level features in the data. However, finding sparse codes remains a very difficult computational problem. In this paper, we present efficient sparse coding algorithms that are based on iteratively solving two convex optimization problems: an L1 -regularized least squares problem and an L2 -constrained least squares problem. We propose novel algorithms to solve both of these optimization problems. Our algorithms result in a significant speedup for sparse coding, allowing us to learn larger sparse codes than possible with previously described algorithms. We apply these algorithms to natural images and demonstrate that the inferred sparse codes exhibit end-stopping and non-classical receptive field surround suppression and, therefore, may provide a partial explanation for these two phenomena in V1 neurons.

🚀 Conference Pioneer — NIPS 2006
📈 Trend Setter — Self-Supervised Learning
🧭 Keyword Pioneer — dictionary learning
🐣 Hot Topic Early Bird — unsupervised learning
🐝 Cross-Pollinator — Artificial Intelligence, Computer Science, Computer Vision, Data Science & Analytics, Deep Learning, Healthcare & Medicine, Interdisciplinary, Knowledge & Reasoning, Machine Learning, Mathematics & Optimization, Natural Language Processing, Reinforcement Learning, Robotics, Security & Privacy, Speech & Audio
🌱 Topic Pioneer — Representation Learning
🌉 Interdisciplinary Bridge — Deep Learning and Interdisciplinary and Machine Learning

Authors

Honglak Lee , Alexis Battle , Rajat Raina , Andrew Y. Ng

Topics

Machine Learning > Core Methods > Representation Learning Machine Learning > Learning Types > Self-Supervised Learning Machine Learning > Learning Types > Unsupervised Learning Machine Learning > Optimization & Theory > Optimization Interdisciplinary > Cognitive Science > Perception Machine Learning > Core Methods > Feature Learning Machine Learning > Core Methods > Optimization Deep Learning > Optimization & Theory > Optimization Deep Learning > Learning Types > Representation Learning Machine Learning > Core Methods > Sparse Coding

Keywords

dictionary learning unsupervised learning feature learning iterative optimization convex optimization feature extraction sparse coding signal representation compressive sensing l1 regularization natural images v1 neurons basis function iterative algorithm
Download PDF

Related papers

Temporal Coding using the Response Properties of Spiking Neurons 2006
Parameter Expanded Variational Bayesian Methods 2006
Effects of Stress and Genotype on Meta-parameter Dynamics in Reinforcement Learning 2006
Ordinal Regression by Extended Binary Classification 2006
Blind source separation for over-determined delayed mixtures 2006