← Back to papers
2022 JMLR JMLR 2022

A Momentumized, Adaptive, Dual Averaged Gradient Method

Abstract

We introduce MADGRAD, a novel optimization method in the family of AdaGrad adaptive gradient methods. MADGRAD shows excellent performance on deep learning optimization problems from multiple fields, including classification and image-to-image tasks in vision, and recurrent and bidirectionally-masked models in natural language processing. For each of these tasks, MADGRAD matches or outperforms both SGD and ADAM in test set performance, even on problems for which adaptive methods normally perform poorly. [abs] [ pdf ][ bib ] [ code ] © JMLR 2022. (edit, beta)

🌉 Interdisciplinary Bridge — Deep Learning and Machine Learning and Mathematics & Optimization
🧭 Keyword Pioneer — deep learning optimization
🐝 Cross-Pollinator — Artificial Intelligence, Computer Science, Computer Vision, Data Science & Analytics, Deep Learning, Machine Learning, Mathematics & Optimization, Reinforcement Learning

Authors

Aaron Defazio , Samy Jelassi

Topics

Machine Learning > Optimization & Theory > Optimization Deep Learning > Architectures > Neural Networks Mathematics & Optimization > Optimization > Continuous Optimization Machine Learning > Learning Types > Deep Learning Deep Learning > Optimization & Theory > Neural Network Optimization

Keywords

stochastic gradient descent deep learning gradient descent dual averaging optimization algorithm adam optimizer adaptive gradient method deep learning optimization adaptive gradient
Download PDF

Related papers

Prior Adaptive Semi-supervised Learning with Application to EHR Phenotyping 2022
LinCDE: Conditional Density Estimation via Lindsey's Method 2022
Causal Classification: Treatment Effect Estimation vs. Outcome Prediction 2022
Provable Tensor-Train Format Tensor Completion by Riemannian Optimization 2022
Power Iteration for Tensor PCA 2022