In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning

12/20/2014

∙

We present experiments demonstrating that some other form of capacity control, different from network size, plays a central role in learning multilayer feed-forward networks. We argue, partially through analogy to matrix factorization, that this is an inductive bias that can help shed light on deep learning.

READ FULL TEXT

In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning

Rethinking Neural vs. Matrix-Factorization Collaborative Filtering: the Theoretical Perspectives

PUFFIN: A Path-Unifying Feed-Forward Interfaced Network for Vapor Pressure Prediction

Feed-Forward Neural Networks Need Inductive Bias to Learn Equality Relations

DeepECMP: Predicting Extracellular Matrix Proteins using Deep Learning

The Inductive Bias of Flatness Regularization for Deep Matrix Factorization

Intrinsic dimensionality and generalization properties of the ℛ-norm inductive bias

Scaling MLPs: A Tale of Inductive Bias

In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning

Related Research

Rethinking Neural vs. Matrix-Factorization Collaborative Filtering: the Theoretical Perspectives

PUFFIN: A Path-Unifying Feed-Forward Interfaced Network for Vapor Pressure Prediction

Feed-Forward Neural Networks Need Inductive Bias to Learn Equality Relations

DeepECMP: Predicting Extracellular Matrix Proteins using Deep Learning

The Inductive Bias of Flatness Regularization for Deep Matrix Factorization

Intrinsic dimensionality and generalization properties of the ℛ-norm inductive bias

Scaling MLPs: A Tale of Inductive Bias