Correcting Predictions for Approximate Bayesian Inference

09/11/2019
by   Tomasz Kuśmierczyk, et al.
Helsingin yliopisto
0

Bayesian models quantify uncertainty and facilitate optimal decision-making in downstream applications. For most models, however, practitioners are forced to use approximate inference techniques that lead to sub-optimal decisions due to incorrect posterior predictive distributions. We present a novel approach that corrects for inaccuracies in posterior inference by altering the decision-making process. We train a separate model to make optimal decisions under the approximate posterior, combining interpretable Bayesian modeling with optimization of direct predictive accuracy in a principled fashion. The solution is generally applicable as a plug-in module for predictive decision-making for arbitrary probabilistic programs, irrespective of the posterior inference strategy. We demonstrate the approach empirically in several problems, confirming its potential.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

02/02/2019

Variational Bayesian Decision-making for Continuous Utilities

Bayesian decision theory outlines a rigorous framework for making optima...
03/18/2013

Generalized Thompson Sampling for Sequential Decision-Making and Causal Inference

Recently, it has been shown how sampling actions from the predictive dis...
11/12/2020

When Does Uncertainty Matter?: Understanding the Impact of Predictive Uncertainty in ML Assisted Decision Making

As machine learning (ML) models are increasingly being employed to assis...
08/06/2019

Bayesian Incremental Inference Update by Re-using Calculations from Belief Space Planning: A New Paradigm

Inference and decision making under uncertainty are key processes in eve...
04/07/2020

Active Recursive Bayesian Inference with Posterior Trajectory Analysis Using α-Divergence

Recursive Bayesian inference (RBI) provides optimal Bayesian latent vari...
06/13/2021

Post-hoc loss-calibration for Bayesian neural networks

Bayesian decision theory provides an elegant framework for acting optima...
01/18/2010

A Monte Carlo Algorithm for Universally Optimal Bayesian Sequence Prediction and Planning

The aim of this work is to address the question of whether we can in pri...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Bayesian inference provides the fundamental basis for modeling uncertainty. The posterior distribution provides a complete summary of what is known about the parameters of a model given some observed data . In particular, the posterior is necessary and sufficient information for making optimal decisions under uncertainty [1].

Figure 1: Bayesian decision theory gives a closed-form analytic rule for making optimal decisions for a given loss, based on the posterior predictive distribution (red). If the same decision rule is used for approximate predictive distribution (blue) based on approximate inference strategy, the numerical decision may be badly off as indicated by the gap

. Our approach learns to correct for this mistake, by replacing the theoretically optimal decision with parametric decision-making module – a neural network – that maps quantiles of the incorrect predictive distribution into optimal decisions.

For predictive models , such as regression or classification, the decisions correspond to committing to individual decision that minimizes the risk , the expectation of a loss over the uncertainty in the model parameters. For example, to minimize the squared loss we should commit to the mean of the predictive distribution , whereas for a loss that penalizes for overestimation of the true value the optimal decision is to report a suitably chosen lower quantile of the predictive distribution. For many losses the Bayes optimal decision is a simple statistic of the predictive distribution (Table 1).

Unfortunately, the decisions obtained by minimizing the risk are optimal only if we have access to the true posterior . For most models we can only obtain an approximation

. The almost universally adopted practice is to plug in the approximation in lieu of the posterior in the predictive distribution and proceed as though the result were the true predictive distribution. Even though this works relatively well for good approximations (e.g., well-converged Markov Chain Monte Carlo sampler), it may fail miserably for bad approximations. Furthermore, even for good approximations it would make sense to account for common biases; e.g., mean-field variational approximations are known to underestimate the posterior variation and hence also the variation in the predictive distribution

[2]. Consequently, using decision rules optimal for the true posterior will have systematic bias as well.

Even though the problem persists in virtually all use of approximate Bayesian inference in decision-making problems, the literature addressing the issue is limited. The best examples build on the concept of loss-calibrated inference. Originally proposed in the context of variational approximation by Lacoste–Julien et al. [3], the core idea is to take into account the eventual decision problem (characterized by some loss) already during inference by altering the learning objective for approximate inference. Recently Cobb et al. [4] and Kuśmierczyk et al. [5] have presented practical algorithms building on this principle, but despite the elegant theory the procedure runs into several practical difficulties: It is computationally heavy, requires knowing the decision loss already during inference, and the empirical improvements are typically meagre. A similar approach has been proposed also for MCMC [6], with similar drawbacks.

We propose an alternative, novel approach for making optimal decisions based on a probabilistic model characterized by an approximate posterior. Instead of modifying the inference procedure as in loss-calibrated inference, we modify the decision making phase by replacing the analytic Bayes optimal decisions with a parametric function tuned to minimize the risk for the approximate posterior at hand. The proposed approach is conceptually simple, can be applied on top of any approximate posterior irrespective of the manner it was obtained, does not require re-computing the posterior approximation for new losses, and it is demonstrated empirically to clearly outperform the loss-calibrated inference approach [5] in some scenarios. When applied on the true posterior the procedure reverts back to the standard Bayes optimal decision.

Our key idea is to couple the Bayesian model with a decision-making module, in practice a neural network, which takes as input a characterization of the predictive distribution and outputs the decision . It mimics optimal Bayesian decision rules by using the same input sufficient for optimal decisions, but is optimized to minimize a particular risk. The principle is illustrated in Figure 1.

Training a flexible mapping from the predictive distribution to the decisions by direct optimization of the empirical risk can, naturally, overfit to the specific data collection. Furthermore, a flexible mapping may, under some conditions, overrule the underlying model, by making predictions not supported by the assumed model. We overcome these issues by presenting a generalized Bayesian inference strategy, building on [7], over the decision-making modules. We provide a justified prior distribution that controls the decision-making module, by regularizing it towards the standard Bayes optimal decision, and demonstrate automatic prior specification using a light bootstrap procedure [8].

Our result is a plug-in tool that is applicable for every Bayesian model designed for predictive tasks. One can use any modeling framework or inference strategy, as long as we have access to samples from the predictive distribution. We demonstrate the approach for standard Bayesian models ranging from matrix factorization to sparse regression, and show improved decisions for poor approximations.

2 Background

To facilitate understanding the rest of the paper, we briefly summarize Bayesian decision theory, the concept of approximate posterior inference, and the standard approach for incorporating approximate posteriors into Bayesian decision making.

2.1 Bayesian Decision Theory

Loss Expression Decision rule
squared
absolute
imbalanced absolute -percentile
tilted -percentile
Table 1: Example losses and their -optimal decisions.

Bayesian decision theory [1]

defines a rigorous framework for decision-making under uncertainty. Given a loss function

and a posterior distribution

of a parametric model conditioned on the dataset

, the Bayes optimal decision minimizes expected posterior loss

In supervised settings, decisions are made for pairs , and the risk is an expectation over unknown data generating distribution . For setups where we do not explicitly model but do assume a model for , we can define conditional risk for using

(1)

where . The optimal decisions for individual data points are then , denoted as -optimal to indicate they are optimal for the true predictive distribution. Similarly, we call (1) the -risk or simply risk. Table 1 presents -optimal decisions for several example losses.

In practice the risk needs to be approximated by empirical risk , based on some data collection, where the distribution of and

is purely empirical. More detailed discussion of Bayesian decision theory for supervised learning can be found in

[3].

2.2 Approximate Inference

Typically, the posterior distribution is analytically intractable, and we need to resort to approximate inference techniques to replace it with a computationally tractable proxy . A wide array of inference tools are available with different trade-offs between computational time and efficiency. Monte Carlo Markov Chain (MCMC) methods approximate with samples from the posterior, with state-of-the-art algorithms using gradient-based information to improve convergence and speed [9]. Distributional approximations, in turn, assume a parametric approximation family and optimize for its parameters to minimize some discrepancy measure between the approximation and the true posterior. This can be done in multitude of ways, the most common strategies being variational approximation [2] and expectation propagation [10, 11]. Distributional approximations often have computational advantage over MCMC, but are biased or inaccurate especially when using poor approximation family [12]

. For example, variational approximation minimizes a Kullback-Leibler divergence between the approximation and the true posterior, and hence underestimates posterior variation

[2].

Probabilistic programming tools, such as Stan [13] and Edward [14] have recently made practical Bayesian modeling easier, by coupling automatic inference engines based on Hamiltonian Monte Carlo [9] or gradient-based variational approximation [15, 16] with easy model specification language. The typical mode of operation today is to specify the model in such a language, and the practitioner may not even care about the approximation strategy being used. This has made Bayesian modeling possible for wider audience, which implies more research is needed on how the models are being used in downstream applications.

2.3 Decision-making for Approximate Inference

We often do not have access to the true posterior to evaluate the -risk in Eq. 1. We instead use the approximation as a proxy for the posterior. Following the nomenclature of Lacoste–Julien et al. [3], we define the -posterior predictive distribution and the -risk

The -optimal decisions minimize the -risk. Despite the name, the -optimal decisions are not optimal with respect to the -risk. Instead, we typically have . Section 3 describes how the discrepancy between and can be mitigated, when only having access to the approximation .

3 Predictive Decision Theory

Our goal is to make good – possibly even optimal – decisions, as measured by the risk under a probabilistic model , when only having access to an approximation of the true posterior . We do this by introducing a parametric decision-making module , a function that takes as input a characterization of the predictive distribution under the approximate posterior and produces as an output the decision . This module can be applied on top of arbitrary probabilistic models.

In the following, we describe the necessary technical elements required for implementing the proposed Predictive Decision Theory (PDT). We first formulate the problem as generalized Bayesian inference over the decision-makers in Section 3.1, and then explain how practical decision-making modules can be implemented in Section 3.2.

3.1 Decision Belief Distributions

To provide justified uncertainty quantification for the decisions, we build on the generalized Bayesian inference framework by Bissiri et al. [7]. They present a coherent procedure for updating beliefs for scenarios where the parameter of interest is connected to observations via arbitrary loss functions, instead of likelihood functions. Their key result is that (using generic notation to avoid confusion with the symbols used in main derivations of our results)

is a valid posterior distribution for any loss that depends on the parameters and the observation , and any constant allowing for calibration of relative impact of the prior and the loss. In other words, we can use exponentiated negative losses in place of likelihoods and still characterize the uncertainty on in a coherent manner.

The framework provides direct basis for decision-problems involving the model parameters itself. We use it for predictive problems instead, for problems characterized by loss defined in terms of predicted quantities. The decision is made based on the predictive distribution that is obtained via marginalization of model parameters . One could think of plugging in the predictive distribution along with the procedure for obtaining in place of loss, but that would make model the impossible to train because of the marginalization required over . Instead, we consider decision belief distributions

and tie decisions for different with parameteric decision-making module (DM) . DM makes decisions on the observed data , is parameterized by , and takes as input the predictive distribution (or an approximation ) from already trained model. This allows us to define the Bayesian update rule for belief distribution of a decision-making module

(2)

where and is some prior defined over the parameters of the DM. The constant allows tuning the compromise, but importantly all values result in valid posterior.

3.2 Decision Maker

We replace the standard decision-making procedure explained in Section 2.3 with parametric function (decision maker) , for which posterior inference is characterized by Eq. (2). For a collection of data instances the log-posterior is

(3)

where is a normalization constant that has no effect on inference and . Empirical risk

appears naturally as a part of the objective. Suitable priors, however, help us to address the issue of generalization, to make sure the decisions are good for the true risk instead of just the empirical one. The formulation defines the basis for full posterior inference, but we focus on finding a single good decision-maker and hence resort to MAP estimation of Eq. (

3) that is less computationally expensive.

Regularization with -optimal Decisions

We start by specifying the prior used for controlling the flexibility of the DM, by building on the assumption that is reasonably good approximation of . When they are identical the -optimal decision corresponds exactly to the -optimal decision , and for small approximation errors we would still expect to find the optimal decision in the neighborhood of the theoretical optimum.

Building on the above principle we define an implicit prior. Instead of placing a prior on the parameters directly, we place a prior on the decisions , which induces a prior on . For the decisions we assume the prior

(4)

where is -optimal decision for

th data point and the variance is set to unit value as it can be subsumed to the

parameter in Eq. (3). Small allows DM to deviate arbitrary far from , whereas large forces DM to emulate -optimal decisions. A good value can be selected, e.g., by cross-validation.

A more flexible prior is obtained by allowing point-specific variation with

This prior requires an automatic procedure for setting and , which we do based on the expected variation caused by the underlying distribution . The decisions are made conditional on the covariates and the total risk averages over the data generating distribution , as briefly explained in Section 2.1. The natural variation in samples produced by the generating distribution provides a reasonable basis for estimating the allowed variation in decisions as well, and to estimate this we apply a bootstrap procedure [8].

With bootstrapping, we draw several new datasets (), using in our empirical experiments. For each dataset, we obtain the posterior approximation and the -optimal decisions , and estimate the parameters for each data point using , . Finally, under independence assumption on , we can define the prior for as

(5)

where for a simpler case and for bootstrap-based prior . The overall strength of the prior is in both cases controlled by .

Representation of the Predictive Distribution

Bayesian decision theory states that the predictive distribution is necessary and sufficient information for making optimal decisions. We retain this assumption when switching to parametric decision-makers, by using the predictive distribution as the sole input for the decision-maker. However, it is rarely available as analytic expression.

A practical and model-independent representation is obtained by sampling a collection of data points from the model under the approximate posterior . Such a collection can be summarized with suitable finite statistic that can then be passed as input for the decision-maker; we use evenly-spaced empirical quantiles, but for example a histogram would work as well. This representation only requires the ability to sample from the predictive distribution, covering essentially all probabilistic models of interest. It could even generalize for simulator-based models that lack closed-form expression for likelihood but still enable sampling from the model [17].

The parameters and influence the richness and accuracy of the input representation, but as illustrated empirically in Section 5.1 the procedure is not very sensitive to the choices. Since computing the representation is cheap we can safely use large , and small

is enough because the decision-makers can implicitly interpolate between the available quantiles if needing more granularity.

Parametric Decision Makers

Most Bayesian optimal decisions are simple summary statistics of the predictive distributions (Table 1), and simplified illustrations of the procedure like Figure 1 intuitively suggest that it may be enough to modify the specific threshold, e.g., by lowering or increasing the quantile for tilted loss to account for the discrepancy between the predictive distributions. Such a decision-maker is easy to implement and has limited room for overfitting, but we will later show empirically (Section 5.1) that it is not sufficient for notably improving the predictions. Instead, we need more flexible functions.

A natural choice for a flexible model-independent decision maker is to use a neural network, interpreted as arbitrary mapping from the predictive distribution to the decision . The decision-maker is parameterized by

, which denotes collectively the set of all network weights. Such modules are easy to implement in modern machine learning platforms, allowing for flexible choice of network architectures. In our experiments, we use a simple feed-forward network with 3 hidden layers (with 20, 20 and 10 nodes) with ReLU activation and Adam optimizer with learning rate

, but note that the specific network details are not important. Instead, we simply need a network that is able to relatively flexibly convert a fixed-dimensional quantile representation of the predictive distribution into a scalar decision.

Figure 2: Matrix Factorization of last.fm data: (Left:) Predictive performance on test data, showing the proposed method clearly outperforms LCVI for tilted losses. (Middle:) Regularization effects on test and training data for squared and tilted (

) loss, showing how the regularization completely removes overfitting. (Right:) Effects of number of samples and predictive distribution representation (number of quantiles) on test data performance for tilted (

) loss.

4 Related Work

Our work lies in the intersection of Bayesian modeling and learning theory. The general principle of minimizing the regularized risk for making decisions that generalize for the data-generating distribution is the cornerstone of standard learning theory (see, e.g., [18]). The current work does not utilize or require results in recent learning theory literature, but we note that there may be interesting connections useful for future work. Our method also loosely relates to work on quantile regression [19] and the training of decision-makers to generate prediction intervals [20]. However, we build on existing Bayesian models and merely use a neural network to correct for approximation error, whereas they use black-box methods for the predictive distribution itself.

The most closely related work is on direct loss-calibration of posterior approximations [3, 6]. We propose an alternative for that approach and will empirically compare the proposed solution against the latest loss-calibration method [5], and hence describe here their approach briefly.

The core idea of loss-calibration is to replace the original learning objective, in case of variational approximation, a lower bound to the marginal likelihood , with an augmented objective that involves a separate term accounting for the loss

The function is the loss transformed to utility [1]. This augmented objective is maximized, typically with an alternating algorithm, with respect to both the parameters of the approximation and the decisions . The decisions influence the approximation and the approximation influence the decisions, and hence the optimization is tightly coupled. This implies that the procedure needs to be carried out separately for all losses of interest and the algorithmic details need to be derived for each approximation strategy separately. Our approach is applied on top of existing approximation and hence does not suffer from these limitations: Once the approximation is available, we can easily optimize for any loss without needing to re-iterate the inference.

5 Experiments

To illustrate and validate the procedure we conduct a series of experiments, using variational approximation as . We first compare the method against the alternative of calibrating the posterior inference to account for the loss for a matrix factorization model, and then demonstrate improved decisions for a sparse regression model and a multilevel model for cases with approximations. We also study practical decisions regarding the input representation and regularization.

5.1 Predictive Performance

We compare our approach to Loss-calibrated Variational Inference (LCVI) by Kuśmierczyk et al. [5], the closest alternative to our method, using the same simplified probabilistic matrix factorization model

they used for demonstrating loss-calibration improves predictions over standard variational inference on a subset of last.fm data [21]. We use their code111https://github.com/tkusmierczyk/lcvi to precisely reproduce the experiment, matching all of their modeling and approximation choices (, , mean-field Gaussian approximation, and log-transformation for the count data).

Figure 2 (left) compares relative reduction of empirical risk (, where is the risk of -optimal decision on standard approximation) on the test data for the same four losses they used. For tilted losses the proposed approach using neural network as decision-maker dramatically outperforms LCVI, reaching risk reduction depending on the parameter of the loss. This demonstrates the proposed approach has clear practical value over the alternative of fine-tuning the approximation itself. The simpler decision-maker that only optimizes for the quantile is unable to improve the decisions, demonstrating that it is preferable to learn a more flexible mapping.

For squared loss even the neural decision-maker performs poorly, due to overfitting. Figure 2 (middle) illustrates how regularizing the decision-maker towards the -optimal decisions using the prior (5) elegantly removes the overfitting. The method never achieves risk reduction, but instead converges to the standard -optimal estimator. Consequently, there are scenarios for which the computationally more expensive LCVI is able to improve the predictions, whereas the proposed method is not.

5.2 Posterior Representation

In practical computation the posterior predictive distribution is presented using empirical quantiles of samples drawn from the predictive distribution (Section 3.2). The choice of the sample size and the number of quantiles naturally influences the representation accuracy. We study the effect of these parameters on the same data and model as above.

Figure 2 (right) compares test data empirical risk averaged of 10 different random initializations for various representations. The main observations are that it is beneficial to use large sample size, and since the computational cost of drawing the samples is negligible compared to other stages of the procedure (fitting the approximation, training the decision-maker) we use in other experiments. The risk improves also when increasing the representation size, and for the other experiments we use quantiles. However, it is worth noting that already very few quantiles are sufficient for improving the risk compared to -optimal decisions (-risk ).

5.3 Handling VI Failures

The main use for the method is in correcting for inaccurate posterior approximations, and hence we conduct two experiments on two separate regression models to validate this. At the same time, we illustrate the bootstrap-based regularization strategy.

Multilevel Models with Poor Approximation

Figure 3: (Top:) Risk for tilted loss () on test subset of radon data. The proposed method with regularization (solid green) is able to significantly reduce the risk over the -optimal baseline (dashed red) for poor approximations. (Bottom:) The method is very insensitive to the regularization parameter ; all choices improve over the -optimal baseline.

Since variational approximations are trained with iterative algorithms, we can construct a controlled experiment by terminating the inference algorithm early and varying the termination point. In general, approximations trained for shorter time are further away from the final approximation and the true posterior.

We use the radon data and the multi-level model

for modeling it [22]. The model is implemented using the publicly available Stan code222http://mc-stan.org/users/documentation/case-studies/radon.html, using the Minnesota subset of the data and also otherwise conforming to their details. The inference is carried out using automatic differentiation variational inference [16], and we split the data randomly into equally sized training and test set.

Figure 3 (top) compares performance of different decision-making strategies (vertical axis) w.r.t quality of the approximation fit controlled by number of training iterations (horizontal axis). Hamiltonian Monte Carlo [9] provides here sufficiently accurate posterior to act as the baseline, and we see that variational approximation converges sufficiently close to the true posterior around iterations, reaching the -risk. The proposed method without regularization achieves relatively good risk already for very poor posterior approximations, but never converges to the optimal decision due to overfitting. The variant regularized with the bootstrap-based prior in Eq. (5) achieves the best of both worlds, reducing the risk for poor approximations but converging to the -optimal decisions when the approximation becomes good. Figure 3 (bottom) plots the risk at iterations (still incorrect approximation) as a function of the regularization parameter , showing that the method is robust to the choice of ; all values within the range improve compared to the -optimal baseline, and for the risk is virtually identical.

Sparse Models with Failure of Convergence

Figure 4: Risk for squared and tilted () loss on test subset of corn data. For three initializations the approximation fails to converge and -optimal decisions have very high risk, but the decision-maker (blue) is able to correct for the failure.

Variational approximations have trouble with multimodal or otherwise complicated posteriors. Yao et al. [12] use models with horseshoe priors [23] for demonstrating this, and we use the same model family to demonstrate how we can still recover good decisions even when this happens. We use the sparse regression model

following closely a classification model proposed by Piironen and Vehtari [24] with public Stan code333https://github.com/yao-yl/Evaluating-Variational-Inference/blob/master/R_code/glm_bernoulli_rhs.stan, but change to Gaussian likelihood suitable for regression problems. We apply the model on the corn data [25], that we randomly split into equally sized training and test subsets.

For this model the quality of the variational approximation is sensitive to random initialization and the stochastic variation during the optimization, so that occasionally the posterior is reasonable whereas for some runs it converges to a very bad solution. Figure 4 illustrates the risk for 10 independent runs with different random seeds, showing that for 7 runs all decision-making strategies are sufficient. However, for 3 of the runs the -optimal decision fails miserably, yet the parametric decision-maker is able to recover essentially -optimal decisions. In other words, we show that the proposed strategy is able to correct for the mistake in the posterior.

5.4 Model Faithfulness

The main goal of our work is to improve the decisions made given a specific probabilistic model, which means the decisions should remain faithful to the underlying model: The decision-maker should correct for mistakes in the posterior approximation but not in model miss-specification, since the users needs to be able to rely on interperations of the model. Next we show the regularization strategy proposed in Section 3.2 achieves this.

Figure 5 shows the predictive distribution of intentionally incorrect model, a linear model for highly non-linear data. Even though the decision-maker only takes as input the predictive distribution (red shaded area), a flexible enough DM learns to ignore the model and returns the upper quantile of the data distribution that minimizes the risk. This can be done because there exists a mapping from the predictive quantiles back to the covariates, and hence the neural network can implicitly use the covariates themselves for predictions. In other words, it can learn to directly map the non-linear data, without conforming to the linear model.

Regularizing with (4) smoothly transitions the decisions towards the -optimal decision, here a linear function of the covariates. Sufficiently strong regularization prevents the flexible DM module from overriding the model.

Figure 5: Without regularization the decision-maker can ignore the underlying linear model (red), but the proposed regularization strategy prevents this and provides predictions that are compatible with the model and only account for approximation error. Here the -optimal decisions are also -optimal, because there is no approximation error.

6 Conclusion

Even though Bayesian decision theory [1] elegantly separates inference from decisions, this separation is constantly being misused by applying decision rules derived for the true posterior to approximations. This may dramatically affect the decisions, typically in a manner that is difficult to notice since there is no access to the true -risk.

Some effort has been made to remedy this, based on directly modifying the inference process to account for the eventual decision loss [3, 4, 5, 6], but these works are computationally costly and have demonstrated only marginal improvements. We proposed the alternative of retaining the original approximation and changing the decision-making process instead, by replacing the analytic decision with parametric decision-making module. This has the significant advantage of being agnostic to the inference strategy being used, making the approach applicable to, e.g., improving predictive accuracy of models for which inference has already been carried out. To demonstrate the generality of our method, we ran several experiments on top of posterior approximations computed by off-the-shelf probabilistic programming system Stan [13], demonstrating improved decisions for cases where the posterior approximation is inaccurate.

While the notion of directly optimizing for empirical risk may feel somewhat unorthodox in a strict Bayesian sense, we presented several technical arguments supporting the validity of the approach. By building on generalized Bayesian inference [7]

we can reformulate reasoning over the decision-makers as justified Bayesian inference, and we provide a prior distribution that allows controlling for how faithful we remain to the underlying model, with automatic procedure for selecting parameters that does not allow significant deviation. While some theoretical questions of interest still remain, we have demonstrated that the quantitative value of predictive Bayesian models can be improved by incorporating flexible decision-makers borrowed from the deep learning literature without compromising the interpretability of Bayesian modeling.

Acknowledgements

The work was supported by Academy of Finland, under grant 1313125, as well as the Finnish Center for Artificial Intelligence (FCAI), a Flagship of the Academy of Finland. We also thank the Finnish Grid and Cloud Infrastructure (urn:nbn:fi:research-infras-2016072533) for providing computational resources.

References

  • [1] James O Berger. Statistical Decision Theory and Bayesian Analysis; 2nd edition. Springer Series in Statistics. Springer, New York, 1985.
  • [2] David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe. Variational Inference: A Review for Statisticians. Journal of the American Statistical Association, 112(518), 2017.
  • [3] Simon Lacoste-Julien, Ferenc Huszár, and Zoubin Ghahramani. Approximate inference for the loss-calibrated Bayesian. In Proceedings of the 14th International Conference on Artificial Intelligence and Statistics, 2011.
  • [4] Adam D Cobb, Stephen J Roberts, and Yarin Gal. Loss-Calibrated Approximate Inference in Bayesian Neural Networks. In Theory of Deep Learning workshop, ICML, 2018.
  • [5] Tomasz Kuśmierczyk, Joseph Sakaya, and Arto Klami. Variational Bayesian Decision-making for Continuous Utilities. arXiv preprint arXiv:1902.00792, 2019.
  • [6] Ehsan Abbasnejad, Justin Domke, and Scott Sanner. Loss-calibrated Monte Carlo Action Selection. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.
  • [7] Pier Giovanni Bissiri, Chris C Holmes, and Stephen G Walker. A general framework for updating belief distributions. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 78(5):1103–1130, 2016.
  • [8] Bradley Efron and Robert J Tibshirani. An introduction to the bootstrap. CRC press, 1994.
  • [9] Matthew D. Hoffman and Andrew Gelman. The no-u-turn sampler: Adaptively setting path lengths in hamiltonian monte carlo. Journal of Machine Learning Research, 15:1593–1623, 2014.
  • [10] Andrew Gelman, Aki Vehtari, Pasi Jylänki, Tuomas Sivula, Dustin Tran, Swupnil Sahai, Paul Blomstedt, John P Cunningham, David Schiminovich, and Christian Robert. Expectation propagation as a way of life: A framework for Bayesian inference on partitioned data. arXiv preprint arXiv:1412.4869, 2017.
  • [11] Thomas P. Minka. Expectation Propagation for Approximate Bayesian Inference. In Proceedings of the 17th Conference on Uncertainty in Artificial Intelligence, 2001.
  • [12] Yuling Yao, Aki Vehtari, Daniel Simpson, and Andrew Gelman. Yes, but did it work?: Evaluating variational inference. In Proceedings of the 35th International Conference on Machine Learning, 2018.
  • [13] Bob Carpenter, Andrew Gelman, Matthew D Hoffman, Daniel Lee, Ben Goodrich, Michael Betancourt, Marcus Brubaker, Jiqiang Guo, Peter Li, and Allen Riddell. Stan: A Probabilistic Programming Language. Journal of Statistical Software, 76(1), 2017.
  • [14] Dustin Tran, Alp Kucukelbir, Adji B. Dieng, Maja Rudolph, Dawen Liang, and David M. Blei. Edward: A Library for Probabilistic Modeling, Inference, and Criticism. arXiv:1610.09787, 2016.
  • [15] Michalis Titsias and Miguel Lázaro-Gredilla. Doubly Stochastic Variational Bayes for non-Conjugate Inference. In Proceedings of 31st International Conference on Machine Learning, 2014.
  • [16] Alp Kucukelbir, Dustin Tran, Rajesh Ranganath, Andrew Gelman, and David M Blei. Automatic Differentiation Variational Inference. The Journal of Machine Learning Research, 18(1), 2017.
  • [17] Michael U. Gutmann and Jukka Corander. Bayesian optimization for likelihood-free inference of simulator-based statistical models. Journal of Machine Learning Research, 17(125):1–47, 2016.
  • [18] Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical Learning: Data mining, inference and prediction. Springer, 2nd edition edition, 2009.
  • [19] Roger Koenker and Kevin F. Hallock. Quantile regression. Journal of Economic Perspectives, 15(4):143–156, December 2001.
  • [20] Tim Pearce, Alexandra Brintrup, Mohamed Zaki, and Andy Neely. High-quality prediction intervals for deep learning: A distribution-free, ensembled approach. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4075–4084, 2018.
  • [21] Thierry Bertin-Mahieux, Daniel P.W. Ellis, Brian Whitman, and Paul Lamere. The Million Song Dataset. In Proceedings of the 12th International Conference on Music Information Retrieval, 2011.
  • [22] Andrew Gelman and Jennifer Hill. Data analysis using regression and multilevel/hierarchical models. Cambridge university press, 2006.
  • [23] Carlos M. Carvalho, Nicholas G. Polson, and James G. Scott. The horseshoe estimator for sparse signals. Biometrika, 97:465–480, 2010.
  • [24] Juho Piironen and Aki Vehtari.

    On the hyperprior choice for the global shrinkage parameter in the horseshoe prior.

    In Artificial Intelligence and Statistics, pages 905–913, 2017.
  • [25] Tao Chen and Elaine Martin.

    Bayesian linear regression and variable selection for spectroscopic calibration.

    Analytica chimica acta, 631(1):13–21, 2009.