← Back to papers
2017 NIPS NeurIPS 2017

Exploring Generalization in Deep Learning

Abstract

With a goal of understanding what drives generalization in deep networks, we consider several recently suggested explanations, including norm-based control, sharpness and robustness. We study how these measures can ensure generalization, highlighting the importance of scale normalization, and making a connection between sharpness and PAC-Bayes theory. We then investigate how well the measures explain different observed phenomena.

🌉 Interdisciplinary Bridge — Deep Learning and Machine Learning
📈 Trend Setter — Robustness
🧭 Keyword Pioneer — sharpness measure
🐣 Hot Topic Early Bird — generalization bound
🐝 Cross-Pollinator — Artificial Intelligence, Computer Science, Computer Vision, Data Science & Analytics, Deep Learning, Healthcare & Medicine, Interdisciplinary, Knowledge & Reasoning, Machine Learning, Mathematics & Optimization, Natural Language Processing, Reinforcement Learning, Robotics, Security & Privacy, Speech & Audio

Authors

Behnam Neyshabur , Srinadh Bhojanapalli , David Mcallester , Nati Srebro

Topics

Machine Learning > Optimization & Theory > Learning Theory Machine Learning > Optimization & Theory > Theory Deep Learning > Learning Types > Representation Learning Machine Learning > Learning Types > Robustness Machine Learning > Learning Types > Generalization

Keywords

pac-bayes theory generalization bound deep network sharpness measure norm-based control
Download PDF

Related papers

High-Order Attention Models for Visual Question Answering 2017
Breaking the Nonsmooth Barrier: A Scalable Parallel Method for Composite Optimization 2017
Premise Selection for Theorem Proving by Deep Graph Embedding 2017
Neural Program Meta-Induction 2017
Safe and Nested Subgame Solving for Imperfect-Information Games 2017