Universal Regular Conditional Distributions
share
We introduce a general framework for approximating regular conditional distributions (RCDs). Our approximations of these RCDs are implemented by a new class of geometric deep learning models with inputs in ℝ^d and outputs in the Wasserstein-1 space 𝒫_1(ℝ^D). We find that the models built using our framework can approximate any continuous functions from ℝ^d to 𝒫_1(ℝ^D) uniformly on compacts, and quantitative rates are obtained. We identify two methods for avoiding the "curse of dimensionality"; i.e.: the number of parameters determining the approximating neural network depends only polynomially on the involved dimension and the approximation error. The first solution describes functions in C(ℝ^d,𝒫_1(ℝ^D)) which can be efficiently approximated on any compact subset of ℝ^d. Conversely, the second approach describes sets in ℝ^d, on which any function in C(ℝ^d,𝒫_1(ℝ^D)) can be efficiently approximated. Our framework is used to obtain an affirmative answer to the open conjecture of Bishop (1994); namely: mixture density networks are universal regular conditional distributions. The predictive performance of the proposed models is evaluated against comparable learning models on various probabilistic predictions tasks in the context of ELMs, model uncertainty, and heteroscedastic regression. All the results are obtained for more general input and output spaces and thus apply to geometric deep learning contexts.
READ FULL TEXT