Rate Optimal Variational Bayesian Inference for Sparse DNN

10/10/2019
by   Jincheng Bai, et al.
0

Sparse deep neural network (DNN) has drawn much attention in recent studies because it possesses a great approximation power and is easier to implement and store in practice comparing to fully connected DNN. In this work, we consider variational Bayesian inference, a computationally efficient alternative to Markov chain Monte Carlo method, on the sparse DNN modeling under spike-and-slab prior. Our theoretical investigation shows that, for any α-Hölder smooth function, the variational posterior distribution shares the (near-)optimal contraction property, and the variation inference leads to (near-)optimal generalization error, as long as the network architecture is properly tuned according to smoothness parameter α. Furthermore, an adaptive variational inference procedure is developed to automatically select optimal network structure even when α is unknown. Our result also applies to the case that the truth is instead a ReLU neural network, and certain contraction bound is obtained.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
08/09/2019

Generalization Error Bounds for Deep Variational Inference

Variational inference is becoming more and more popular for approximatin...
research
08/09/2019

Convergence Rates of Variational Inference in Sparse Deep Learning

Variational inference is becoming more and more popular for approximatin...
research
10/09/2019

Practical Posterior Error Bounds from Variational Objectives

Variational inference has become an increasingly attractive, computation...
research
03/24/2018

Posterior Concentration for Sparse Deep Learning

Spike-and-Slab Deep Learning (SS-DL) is a fully Bayesian alternative to ...
research
04/29/2021

What Are Bayesian Neural Network Posteriors Really Like?

The posterior over Bayesian neural network (BNN) parameters is extremely...
research
06/12/2022

Variational Bayes Deep Operator Network: A data-driven Bayesian solver for parametric differential equations

Neural network based data-driven operator learning schemes have shown tr...
research
03/04/2018

Deep Network Regularization via Bayesian Inference of Synaptic Connectivity

Deep neural networks (DNNs) often require good regularizers to generaliz...

Please sign up or login with your details

Forgot password? Click here to reset