← Back to papers
2018 NIPS NeurIPS 2018

On Neuronal Capacity

Abstract

We define the capacity of a learning machine to be the logarithm of the number (or volume) of the functions it can implement. We review known results, and derive new results, estimating the capacity of several neuronal models: linear and polynomial threshold gates, linear and polynomial threshold gates with constrained weights (binary weights, positive weights), and ReLU neurons. We also derive capacity estimates and bounds for fully recurrent networks and layered feedforward networks.

🌉 Interdisciplinary Bridge — Deep Learning and Machine Learning
🧭 Keyword Pioneer — function implementation
🐝 Cross-Pollinator — Artificial Intelligence, Computer Science, Computer Vision, Data Science & Analytics, Deep Learning, Healthcare & Medicine, Interdisciplinary, Machine Learning, Mathematics & Optimization, Natural Language Processing, Reinforcement Learning, Robotics, Speech & Audio

Authors

Pierre Baldi , Roman Vershynin

Topics

Machine Learning > Optimization & Theory > Learning Theory Machine Learning > Optimization & Theory > Theory Deep Learning > Optimization & Theory > Neural Network Optimization Deep Learning > Optimization & Theory > Theory

Keywords

feedforward network recurrent network function implementation relu neuron neural network capacity neuronal capacity polynomial threshold gate network capacity threshold gate capacity bound
Download PDF

Related papers

Maximum Causal Tsallis Entropy Imitation Learning 2018
Recurrent World Models Facilitate Policy Evolution 2018
Bandit Learning in Concave N-Person Games 2018
Algorithmic Assurance: An Active Approach to Algorithmic Testing using Bayesian Optimisation 2018
PAC-Bayes bounds for stable algorithms with instance-dependent priors 2018