← Back to papers
2023 ICML ICML 2023

Global optimality of Elman-type RNNs in the mean-field regime

Abstract

We analyze Elman-type recurrent neural networks (RNNs) and their training in the mean-field regime. Specifically, we show convergence of gradient descent training dynamics of the RNN to the corresponding mean-field formulation in the large width limit. We also show that the fixed points of the limiting infinite-width dynamics are globally optimal, under some assumptions on the initialization of the weights. Our results establish optimality for feature-learning with wide RNNs in the mean-field regime.

🌉 Interdisciplinary Bridge — Deep Learning and Machine Learning
🧭 Keyword Pioneer — elman recurrent neural network
🐝 Cross-Pollinator — Artificial Intelligence, Computer Science, Computer Vision, Data Science & Analytics, Deep Learning, Healthcare & Medicine, Interdisciplinary, Knowledge & Reasoning, Machine Learning, Mathematics & Optimization, Natural Language Processing, Reinforcement Learning, Robotics, Security & Privacy, Speech & Audio

Authors

Andrea Agazzi , Jianfeng Lu , Sayan Mukherjee

Topics

Machine Learning > Optimization & Theory > Theory Deep Learning > Architectures > Neural Networks Deep Learning > Optimization & Theory > Theory

Keywords

feature learning gradient descent recurrent neural network global optimality infinite-width limit mean-field regime elman recurrent neural network
Download PDF

Related papers

Orthogonality-Enforced Latent Space in Autoencoders: An Approach to Learning Disentangled Representations 2023
On the Convergence Rate of Gaussianization with Random Rotations 2023
Synthetic Prompting: Generating Chain-of-Thought Demonstrations for Large Language Models 2023
Generative Adversarial Symmetry Discovery 2023
Function-Space Regularization in Neural Networks: A Probabilistic Perspective 2023