On Compression Principle and Bayesian Optimization for Neural Networks

06/23/2020
by   Michael Tetelman, et al.
13

Finding methods for making generalizable predictions is a fundamental problem of machine learning. By looking into similarities between the prediction problem for unknown data and the lossless compression we have found an approach that gives a solution. In this paper we propose a compression principle that states that an optimal predictive model is the one that minimizes a total compressed message length of all data and model definition while guarantees decodability. Following the compression principle we use Bayesian approach to build probabilistic models of data and network definitions. A method to approximate Bayesian integrals using a sequence of variational approximations is implemented as an optimizer for hyper-parameters: Bayesian Stochastic Gradient Descent (BSGD). Training with BSGD is completely defined by setting only three parameters: number of epochs, the size of the dataset and the size of the minibatch, which define a learning rate and a number of iterations. We show that dropout can be used for a continuous dimensionality reduction that allows to find optimal network dimensions as required by the compression principle.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
11/30/2019

Learning Rate Dropout

The performance of a deep neural network is highly dependent on its trai...
research
05/21/2021

Escaping Saddle Points with Compressed SGD

Stochastic gradient descent (SGD) is a prevalent optimization technique ...
research
06/25/2020

Automatic Tuning of Stochastic Gradient Descent with Bayesian Optimisation

Many machine learning models require a training procedure based on runni...
research
11/10/2021

Self-Compression in Bayesian Neural Networks

Machine learning models have achieved human-level performance on various...
research
03/15/2021

Data Discovery Using Lossless Compression-Based Sparse Representation

Sparse representation has been widely used in data compression, signal a...
research
12/18/2018

Entropy-Constrained Training of Deep Neural Networks

We propose a general framework for neural network compression that is mo...
research
04/28/2011

Notes on a New Philosophy of Empirical Science

This book presents a methodology and philosophy of empirical science bas...

Please sign up or login with your details

Forgot password? Click here to reset