DeepAI AI Chat
Log In Sign Up

Distilling the Knowledge from Normalizing Flows

by   Dmitry Baranchuk, et al.

Normalizing flows are a powerful class of generative models demonstrating strong performance in several speech and vision problems. In contrast to other generative models, normalizing flows are latent variable models with tractable likelihoods and allow for stable training. However, they have to be carefully designed to represent invertible functions with efficient Jacobian determinant calculation. In practice, these requirements lead to overparameterized and sophisticated architectures that are inferior to alternative feed-forward models in terms of inference time and memory consumption. In this work, we investigate whether one can distill flow-based models into more efficient alternatives. We provide a positive answer to this question by proposing a simple distillation approach and demonstrating its effectiveness on state-of-the-art conditional flow-based models for image super-resolution and speech synthesis.


page 12

page 13


f-VAEs: Improve VAEs with Conditional Flows

In this paper, we integrate VAEs and flow-based generative models succes...

Wavelet Flow: Fast Training of High Resolution Normalizing Flows

Normalizing flows are a class of probabilistic generative models which a...

PassFlow: Guessing Passwords with Generative Flows

Recent advances in generative machine learning models rekindled research...

Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design

Flow-based generative models are powerful exact likelihood models with e...

Scaling Up Probabilistic Circuits by Latent Variable Distillation

Probabilistic Circuits (PCs) are a unified framework for tractable proba...

A mean-field games laboratory for generative modeling

In this paper, we demonstrate the versatility of mean-field games (MFGs)...

Attentive Contractive Flow: Improved Contractive Flows with Lipschitz-constrained Self-Attention

Normalizing flows provide an elegant method for obtaining tractable dens...