Learnable Bernoulli Dropout for Bayesian Deep Learning

02/12/2020
by   Shahin Boluki, et al.
0

In this work, we propose learnable Bernoulli dropout (LBD), a new model-agnostic dropout scheme that considers the dropout rates as parameters jointly optimized with other model parameters. By probabilistic modeling of Bernoulli dropout, our method enables more robust prediction and uncertainty quantification in deep models. Especially, when combined with variational auto-encoders (VAEs), LBD enables flexible semi-implicit posterior representations, leading to new semi-implicit VAE (SIVAE) models. We solve the optimization for training with respect to the dropout parameters using Augment-REINFORCE-Merge (ARM), an unbiased and low-variance gradient estimator. Our experiments on a range of tasks show the superior performance of our approach compared with other commonly used dropout schemes. Overall, LBD leads to improved accuracy and uncertainty estimates in image classification and semantic segmentation. Moreover, using SIVAE, we can achieve state-of-the-art performance on collaborative filtering for implicit feedback on several public datasets.

READ FULL TEXT
research
06/08/2015

Variational Dropout and the Local Reparameterization Trick

We investigate a local reparameterizaton technique for greatly reducing ...
research
11/30/2018

Evaluating Bayesian Deep Learning Methods for Semantic Segmentation

Deep learning has been revolutionary for computer vision and semantic se...
research
01/19/2017

Variational Dropout Sparsifies Deep Neural Networks

We explore a recently proposed Variational Dropout technique that provid...
research
01/07/2021

A Novel Regression Loss for Non-Parametric Uncertainty Optimization

Quantification of uncertainty is one of the most promising approaches to...
research
03/06/2021

Contextual Dropout: An Efficient Sample-Dependent Dropout Module

Dropout has been demonstrated as a simple and effective module to not on...
research
04/09/2019

L_0-ARM: Network Sparsification via Stochastic Binary Optimization

We consider network sparsification as an L_0-norm regularized binary opt...
research
11/01/2021

Comparing Bayesian Models for Organ Contouring in Headand Neck Radiotherapy

Deep learning models for organ contouring in radiotherapy are poised for...

Please sign up or login with your details

Forgot password? Click here to reset