← Back to papers
2018 NIPS NeurIPS 2018

Implicit Bias of Gradient Descent on Linear Convolutional Networks

Abstract

We show that gradient descent on full-width linear convolutional networks of depth $L$ converges to a linear predictor related to the $\ell_{2/L}$ bridge penalty in the frequency domain. This is in contrast to linearly fully connected networks, where gradient descent converges to the hard margin linear SVM solution, regardless of depth.

🌉 Interdisciplinary Bridge — Deep Learning and Machine Learning
🧭 Keyword Pioneer — implicit bia
🐣 Hot Topic Early Bird — implicit bia
🐝 Cross-Pollinator — Artificial Intelligence, Computer Science, Computer Vision, Data Science & Analytics, Deep Learning, Healthcare & Medicine, Interdisciplinary, Knowledge & Reasoning, Machine Learning, Mathematics & Optimization, Natural Language Processing, Reinforcement Learning, Robotics, Security & Privacy, Speech & Audio

Authors

Suriya Gunasekar , Jason Lee , Daniel Soudry , Nati Srebro

Topics

Machine Learning > Optimization & Theory > Learning Theory Machine Learning > Optimization & Theory > Neural Network Optimization Machine Learning > Optimization & Theory > Theory Deep Learning > Optimization & Theory > Neural Network Optimization Deep Learning > Optimization & Theory > Theory

Keywords

learning theory gradient descent implicit bia frequency domain linear convolutional network bridge penalty
Download PDF

Related papers

Maximum Causal Tsallis Entropy Imitation Learning 2018
Recurrent World Models Facilitate Policy Evolution 2018
Bandit Learning in Concave N-Person Games 2018
Algorithmic Assurance: An Active Approach to Algorithmic Testing using Bayesian Optimisation 2018
PAC-Bayes bounds for stable algorithms with instance-dependent priors 2018