← Back to papers
2021 JMLR JMLR 2021

When Does Gradient Descent with Logistic Loss Find Interpolating Two-Layer Networks?

Abstract

We study the training of finite-width two-layer smoothed ReLU networks for binary classification using the logistic loss. We show that gradient descent drives the training loss to zero if the initial loss is small enough. When the data satisfies certain cluster and separation conditions and the network is wide enough, we show that one step of gradient descent reduces the loss sufficiently that the first result applies. [abs] [ pdf ][ bib ] © JMLR 2021. (edit, beta)

❓ The Questioner
🧭 Keyword Pioneer — interpolating network
🐝 Cross-Pollinator — Artificial Intelligence, Computer Science, Data Science & Analytics, Deep Learning, Healthcare & Medicine, Machine Learning, Mathematics & Optimization, Natural Language Processing, Reinforcement Learning, Speech & Audio
🌉 Interdisciplinary Bridge — Deep Learning and Machine Learning

Authors

Niladri S. Chatterji , Philip M. Long , Peter L. Bartlett

Topics

Machine Learning > Optimization & Theory > Learning Theory Machine Learning > Optimization & Theory > Neural Network Optimization Machine Learning > Learning Types > Supervised Learning Machine Learning > Learning Types > Representation Learning Deep Learning > Optimization & Theory > Optimization

Keywords

binary classification neural network optimization gradient descent logistic loss two-layer network interpolating network
Download PDF

Related papers

Optimal Feedback Law Recovery by Gradient-Augmented Sparse Polynomial Regression 2021
Normalizing Flows for Probabilistic Modeling and Inference 2021
Determining the Number of Communities in Degree-corrected Stochastic Block Models 2021
Guided Visual Exploration of Relations in Data Sets 2021
Safe Policy Iteration: A Monotonically Improving Approximate Policy Iteration Approach 2021