DeepAI
Log In Sign Up

Deep discriminative to kernel generative modeling

01/31/2022
by   Jayanta Dey, et al.
12

The fight between discriminative versus generative goes deep, in both the study of artificial and natural intelligence. In our view, both camps have complementary value, so, we sought to synergistic combine them. Here, we propose a methodology to convert deep discriminative networks to kernel generative networks. We leveraged the fact that deep models, including both random forests and deep networks, learn internal representations which are unions of polytopes with affine activation functions to conceptualize them both as generalized partitioning rules. From that perspective, we used foundational results on the relationship between histogram rules and kernel density estimators to obtain class conditional kernel density estimators from the deep models. We then studied the trade-offs we observed from implementing this strategy in low-dimensional settings, both theoretically and empirically, as a first step towards understanding. Theoretically, we show conditions under which our generative models are more efficient than the corresponding discriminative approaches. Empirically, when sample sizes are relatively high, the discriminative models tend to perform as well or better on discriminative metrics, such as classification rates and posterior calibration. However, when sample sizes are relatively low, the generative models outperform the discriminative ones even on discriminative metrics. Moreover, the generative ones can also sample from the distribution, obtain smoother posteriors, and extrapolate beyond the convex hull of the training data to handle OOD inputs more reasonably. Via human experiments we illustrate that our kernel generative networks (Kragen) behave more like humans than deep discriminative networks. We believe this approach may be an important step in unifying the thinking and approaches across the discriminative and generative divide.

READ FULL TEXT

page 7

page 10

09/19/2011

Learning Discriminative Metrics via Generative Models and Kernel Learning

Metrics specifying distances between data points can be learned in a dis...
03/06/2017

Generative and Discriminative Text Classification with Recurrent Neural Networks

We empirically characterize the performance of discriminative and genera...
02/15/2013

Density Ratio Hidden Markov Models

Hidden Markov models and their variants are the predominant sequential c...
01/08/2020

Learning Generative Models using Denoising Density Estimators

Learning generative probabilistic models that can estimate the continuou...
04/02/2019

Generative-Discriminative Complementary Learning

Majority of state-of-the-art deep learning methods for vision applicatio...
12/01/2021

Forward Operator Estimation in Generative Models with Kernel Transfer Operators

Generative models which use explicit density modeling (e.g., variational...
06/25/2020

Joints in Random Forests

Decision Trees (DTs) and Random Forests (RFs) are powerful discriminativ...