← Back to papers
2020 JMLR JMLR 2020

Convergence Rates for the Stochastic Gradient Descent Method for Non-Convex Objective Functions

Abstract

We prove the convergence to minima and estimates on the rate of convergence for the stochastic gradient descent method in the case of not necessarily locally convex nor contracting objective functions. In particular, the analysis relies on a quantitative use of mini-batches to control the loss of iterates to non-attracted regions. The applicability of the results to simple objective functions arising in machine learning is shown. [abs] [ pdf ][ bib ] © JMLR 2020. (edit, beta)

🌉 Interdisciplinary Bridge — Machine Learning and Mathematics & Optimization
🐣 Hot Topic Early Bird — stochastic gradient descent
🐝 Cross-Pollinator — Artificial Intelligence, Computer Science, Computer Vision, Data Science & Analytics, Deep Learning, Knowledge & Reasoning, Machine Learning, Mathematics & Optimization, Reinforcement Learning, Security & Privacy

Authors

Benjamin Fehrman , Benjamin Gess , Arnulf Jentzen

Topics

Machine Learning > Optimization & Theory > Neural Network Optimization Machine Learning > Optimization & Theory > Stochastic Processes Mathematics & Optimization > Optimization > Stochastic Methods Machine Learning > Core Methods > Optimization Deep Learning > Optimization & Theory > Optimization

Keywords

stochastic gradient descent non-convex optimization convergence rate objective function
Download PDF

Related papers

Weighted Message Passing and Minimum Energy Flow for Heterogeneous Stochastic Block Models with Side Information 2020
Connecting Spectral Clustering to Maximum Margins and Level Sets 2020
Stochastic Nested Variance Reduction for Nonconvex Optimization 2020
Gradient Descent for Sparse Rank-One Matrix Completion for Crowd-Sourced Aggregation of Sparsely Interacting Workers 2020
A Sparse Semismooth Newton Based Proximal Majorization-Minimization Algorithm for Nonconvex Square-Root-Loss Regression Problems 2020