Does the Geometry of the Data Control the Geometry of Neural Predictions? (Student Abstract)

Abstract This paper studies the over-parameterization of deep neural networks using the Fisher Information Matrix from information geometry. We identify several surprising trends in the structure of its eigenspectrum, and how this structure relates to the eigenspectrum of the data correlation matrix. We identify how the eigenspectrum relates to the topology of the predictions of the model and develop a "model reduction'' method for deep networks. This ongoing investigation hypothesizes certain universal trends in the FIM of deep networks that may shed light on their effectiveness.