Implicit Normalizing Flows

1 Introduction

Normalizing flows (NFs) (rezende2015variational; nice) are promising methods for density modeling. NFs define a model distribution $p_{x} (x)$ by specifying an invertible transformation $f (x)$ from $x$

to another random variable

z

. By change-of-variable formula, the model density is

ln p_{x} (x) = ln p_{z} (f (x)) + ln ∣ ∣ det (J_{f} (x)) ∣ ∣,

(1)

where $p_{z} (z)$ follows a simple distribution, such as Gaussian. NFs are particularly attractive due to their tractability, i.e., the model density $p_{x} (x)$ can be directly evaluated as Eqn. (1). To achieve such tractability, NF models should satisfy two requirements: (i) the mapping between $x$ and $z$ is invertible; (ii) the log-determinant of the Jacobian $J_{f} (x)$ is tractable. Searching for rich model families that satisfy these tractability constraints is crucial for the advance of normalizing flow research. For the second requirement, earlier works such as inverse autoregressive flow (kingma2016improved) and RealNVP (realnvp) restrict the model family to those with triangular Jacobian matrices.

More recently, there emerge some free-form Jacobian approaches, such as Residual Flows (ResFlows) (behrmann2019invertible; chen2019residual)

. They relax the triangular Jacobian constraint by utilizing a stochastic estimator of the log-determinant, enriching the model family. However, the Lipschitz constant of each transformation block is constrained for invertibility. In general, this is not preferable because mapping a simple prior distribution to a potentially complex data distribution may require a transformation with a very large Lipschitz constant (See Fig.

3 for a 2D example). Moreover, all the aforementioned methods assume that there exists an explicit forward mapping

z = f (x)

. Bijections with explicit forward mapping only covers a fraction of the broad class of invertible functions suggested by the first requirement, which may limit the model capacity.

In this paper, we propose implicit flows (ImpFlows) to generalize NFs, allowing the transformation to be implicitly defined by an equation $F (z, x) = 0$ . Given $x$ (or $z$ ), the other variable can be computed by an implicit root-finding procedure $z = R o o t F i n d (F (\cdot, x))$ . An explicit mapping $z = f (x)$ used in prior NFs can viewed as a special case of ImpFlows in the form of $F (z, x) = f (x) - z = 0$ . To balance between expressiveness and tractability, we present a specific from of ImpFlows, where each block is the composition of a ResFlow block and the inverse of another ResFlow block. We theoretically study the model capacity of ResFlows and ImpFlows in the function space. We show that the function family of single-block ImpFlows is strictly richer than that of two-block ResFlows by relaxing the Lipschitz constraints. Furthermore, for any ResFlow with a fixed number of blocks, there exists some invertible function that ResFlow has non-negligible approximation error, but ImpFlow can exactly model.

On the practical side, we develop a scalable algorithm to estimate the probability density and its gradients, and draw samples from ImpFlows. The algorithm leverages the implicit differentiation formula. Despite being more powerful, the gradient computation of ImpFlow is mostly similar with that of ResFlows, except some additional overhead on root finding. We test the effectiveness of ImpFlow on several classification and generative modeling tasks. ImpFlow outperforms ResFlow on all the benchmarks, with comparable model sizes and computational cost.

2 Related Work

Expressive Normalizing Flows There are many works focusing on improving the capacity of NFs. For example, nice; realnvp; kingma2018glow; ho2019flow++; song2019mintnet; hoogeboom2019emerging; de2020block; durkan2019neural design dedicated model architectures with tractable Jacobian. More recently, grathwohl2018ffjord; behrmann2019invertible; chen2019residual propose NFs with free-form Jacobian, which approximate the determinant with stochastic estimators. In parallel with architecture design, chen2020vflow; huang2020augmented; cornish2019relaxing; nielsen2020survae improve the capacity of NFs by operating in a higher-dimensional space. As mentioned in the introduction, all these existing works adopt explicit forward mappings, which is only a subset of the broad class of invertible functions. In contrast, the implicit function family we consider is richer. While we primarily discuss the implicit generalization of ResFlows (chen2019residual) in this paper, the general idea of utilizing implicit invertible functions could be potentially applied to other models as well. Finally, zhang2020approximation formally prove that the model capacity of ResFlows is restricted by the dimension of the residual blocks. In comparison, we study another limitation of ResFlows in terms of the bounded Lipschitz constant, and compare the function family of ResFlows and ImpFlows with a comparable depth.

Continuous Time Flows (CTFs) (chen2018neural; grathwohl2018ffjord; chen2018continuous)

are flexible alternative to discrete time flows for generative modeling. They typically treat the invertible transformation as a dynamical system, which is approximately simulated by ordinary differential equation (ODE) solvers. In contrast, the implicit function family considered in this paper does not contain differential equations, and only requires fixed point solvers. Moreover, the theoretical guarantee is different. While CTFs typically study the universal approximation capacity under the continuous time case (i.e., “

infinite depth” limit), we consider the model capacity of ImpFlows and ResFlows under a finite number of transformation steps. Finally, while CTFs are flexible, their learning is challenging due to instability (liu2020does; massaroli2020stable) and exceedingly many ODE solver steps (finlay2020train), making their large-scale application still an open problem.

Implicit Deep Learning

Utilizing implicit functions enhances the flexibility of neural networks, enabling the design of network layers in a problem-specific way. For instance,

bai2019deep propose a deep equilibrium model as a compact replacement of recurrent networks; amos2017optnet generalize each layer to solve an optimization problem; wang2019satnet integrate logical reasoning into neural networks; reshniak2019robust utilize the implicit Euler method to improve the stability of both forward and backward processes for residual blocks; and sitzmann2020implicit incorporate periodic functions for representation learning. Different from these works, which consider implicit functions as a replacement to feed-forward networks, we develop invertible implicit functions for normalizing flows, discuss the conditions of the existence of such functions, and theoretically study the model capacity of our proposed ImpFlow in the function space.

3 Implicit Normalizing Flows

We now present implicit normalizing flows, by starting with a brief overview of existing work.

3.1 Normalizing Flows

As shown in Eqn. (1), a normalizing flow $f : x \mapsto z$ is an invertible function that defines a probability distribution with the change-of-variable formula. The modeling capacity of normalizing flows depends on the expressiveness of the invertible function $f$ . Residual flows (ResFlows) (chen2019residual; behrmann2019invertible) are a particular powerful class of NFs due to their free-form Jacobian. ResFlows use $f = f_{L} \circ \dots \circ f_{1}$ to construct the invertible mapping, where each layer $f_{l}$ is an invertible residual network with Lipschitz constraints bounded by a fixed constant $κ$ :

f_{l} (x) = x + g_{l} (x), L i p (g_{l}) \leq κ < 1,

(2)

where $L i p (g)$ is the Lipschitz constant of a function $g$ (see Sec. 4.1 for details). Despite their free-form Jacobian, the model capacity of ResFlows is still limited by the Lipschitz constant of the invertible function. The Lipschitz constant of each ResFlow block $f_{l}$ cannot exceed 2 (behrmann2019invertible), so the Lipschitz constant of an $L$ -block ResFlow cannot exceed $2^{L}$ . However, to transfer a simple prior distribution to a potentially complex data distribution, the Lipschitz constant of the transformation can be required to be sufficiently large in general. Therefore, ResFlows can be undesirably deep simply to meet the Lipschitz constraints (see Fig. 3 for a 2D example). Below, we present implicit flows (ImpFlows) to relax the Lipschitz constraints.

3.2 Model Specification

In general, an implicit flow (ImpFlow) is defined as an invertible mapping between random variables $x$ and $z$ of dimension $d$ by finding the roots of $F (z, x) = 0$ , where $F$ is a function from $R^{2 d}$ to $R^{d}$ . In particular, the explicit mappings $z = f (x)$ used in prior flow instances (chen2019residual; kingma2018glow) can be expressed as an implicit function in the form $F (z, x) = f (x) - z = 0$ . While ImpFlows are a powerful family to explore, generally they are not guaranteed to satisfy the invertibility and the tractability of the log-determinant as required by NFs. In this paper, we focus on the following specific form, which achieves a good balance between expressiveness and tractability, and leave other possibilities for future studies.

Definition 1.

Let $g_{z} : R^{d} \to R^{d}$ and $g_{x} : R^{d} \to R^{d}$ be two functions such that $L i p (g_{x}) < 1$ and $L i p (g_{z}) < 1$ , where $L i p (g)$ is the Lipschitz constant of a function $g$ . A specific form of ImpFlows is defined by

F (z, x) = 0, where F (z, x) = g_{x} (x) - g_{z} (z) + x - z .

(3)

The root pairs of Eqn. (3) form a subset in $R^{d} \times R^{d}$ , which actually defines the assignment rule of a unique invertible function $f$ . To see this, for any $x_{0}$ , according to Definition 1, we can construct a contraction with a unique fixed point, which corresponds to a unique root (w.r.t. $z$ ) of $F (z, x_{0}) = 0$ , denoted by $f (x_{0})$ . Similarly, in the reverse process, given a $z_{0}$ , the root (w.r.t. $x$ ) of $F (z_{0}, x) = 0$ also exists and is unique, denoted by $f^{- 1} (z_{0})$ . These two properties are sufficient to ensure the existence and the invertibility of $f$ , as summarized in the following theorem.

Theorem 1.

Eqn.(3) defines a unique mapping $f : R^{d} \to R^{d}, z = f (x)$ , and $f$ is invertible.

See proof in Appendix A.1. Theorem 1 characterizes the validness of the ImpFlows introduced in Definition 1. In fact, a single ImpFlow is a stack of a single ResFlow and the inverse of another single ResFlow, which will be formally stated in Sec 4. We will investigate the expressiveness of the function family of the ImpFlows in Sec 4, and present a scalable algorithm to learn a deep generative model built upon ImpFlows in Sec. 5.

Figure 1: An illustration of our main theoretical results on the expressiveness power of ImpFlows and ResFlows. Panel (a) and Panel (b) correspond to results in Sec. 4.2 and Sec. 4.3 respectively.

4 Expressiveness Power

We first present some preliminaries on Lipschitz continuous functions in Sec. 4.1 and then formally study the expressiveness power of ImpFlows, especially in comparison to ResFlows. In particular, we prove that the function space of ImpFlows is strictly richer than that of ResFlows in Sec. 4.2 (see an illustration in Fig. 1 (a)). Furthermore, for any ResFlow with a fixed number of blocks, there exists some function that ResFlow has a non-negligible approximation error. However, the function is exactly representable by a single-block ImpFlow. The results are illustrated in Fig. 1 (b) and formally presented in Sec. 4.3.

4.1 Lipschitz Continuous Functions

For any differentiable function $f : R^{d} \to R^{d}$ and any $x \in R^{d}$ , we denote the Jacobian matrix of $f$ at $x$ as $J_{f} (x) \in R^{d \times d}$ .

Definition 2.

A function $R^{d} \to R^{d}$ is called Lipschitz continuous if there exists a constant $L$ , s.t.

∥ f (x_{1}) - f (x_{2}) ∥ \leq L ∥ x_{1} - x_{2} ∥, \forall x_{1}, x_{2} \in R^{d} .

The smallest $L$ that satisfies the inequality is called the Lipschitz constant of $f$ , denoted as $L i p (f)$ .

Generally, the definition of $L i p (f)$ depends on the choice of the norm $| | \cdot | |$ , while we use $L_{2}$ -norm by default in this paper for simplicity.

Definition 3.

A function $R^{d} \to R^{d}$ is called bi-Lipschitz continuous if it is Lipschitz continuous and has an inverse mapping $f^{- 1}$ which is also Lipschitz continuous.

It is useful to consider an equivalent definition of the Lipschitz constant in our following analysis.

Proposition 1.

(Rademacher (federer1969grundlehren, Theorem 3.1.6)) If $f : R^{d} \to R^{d}$ is Lipschitz continuous, then $f$ is differentiable almost everywhere, and

L i p (f) = sup x \in R^{d} ∥ J_{f} (x) ∥_{2},

where $∥ M ∥_{2} = {sup}_{{v : ∥ v ∥_{2} = 1}} ∥ M v ∥_{2}$ is the operator norm of the matrix $M \in R^{d \times d}$ .

Figure 2: A 1-D motivating example. (a) Plot of the target function. (b) Results of fitting the target function using ResFlows with different number of blocks. All functions have non-negligible approximation error due to the Lipschtiz constraint. (c) An ImpFlow that can exactly represent the target function. (d) A visualization of compositing a ResFlow block and the inverse of another ResFlow block to construct an ImpFlow block. The detailed settings can be found in Appendix D.

4.2 Comparison to two-block ResFlows

We formally compare the expressive power of a single-block ImpFlow and a two-block ResFlow. We highlight the structure of the theoretical results in this subsection in Fig. 1 (a) and present a 1D motivating example in Fig. 2. All the proofs can be found in Appendix. A.

On the one hand, according to the definition of ResFlow, the function family of the single-block ResFlow is

R

\coloneqq{f:f=g+Id, g∈C1(Rd,Rd),Lip(g)<1},

(4)

where $C^{1} (R^{d}, R^{d})$ consists of all functions from $R^{d}$ to $R^{d}$ with continuous derivatives and $I d$ denotes the identity map. Besides, the function family of $ℓ$ -block ResFlows is defined by composition:

Rℓ\coloneqq{f:f=fℓ∘⋯∘f1 for some% f1,⋯,fℓ∈R}.

(5)

By definition of Eqn. (4) and Eqn. (5), $R_{1} = R$ .

On the other hand, according to the definition of the ImpFlow in Eqn. (3), we can obtain $(g_{x} + I d) (x) = g_{x} (x) + x = g_{z} (z) + z = (g_{z} + I d) (z),$ where $\circ$ denotes the composition of functions. Equivalently, we have $z = ((g_{z} + I d)^{- 1} \circ (g_{x} + I d)) (x),$ which implies the function family of the single-block ImpFlow is

I = {f : f = f_{2}^{- 1} \circ f_{1} for some f_{1}, f_{2} \in R} .

(6)

Intuitively, a single-block ImpFlow can be interpreted as the composition of a ResFlow block and the inverse function of another ResFlow block, which may not have an explicit form (see Fig. 2 (c) and (d) for a 1D example). Therefore, it is natural to investigate the relationship between $I$ and $R_{2}$ . Before that, we first introduce a family of “monotonically increasing functions” that does not have an explicit Lipschitz constraint, and show that it is strictly larger than $R$ .

Lemma 1.

R⫋F\coloneqq{f∈D:infx∈Rd,v∈Rd,∥v∥2=1vTJf(x)v>0},

(7)

where $D$ is the set of all bi-Lipschitz $C^{1}$ -diffeomorphisms from $R^{d}$ to $R^{d}$ , and $A ⫋ B$ means $A$ is a proper subset of $B$ .

Note that it follows from behrmann2019invertible that all functions in $R$ are bi-Lipschitz, so $R ⫋ D$ . In the 1D input case, we can get $R = {f \in C^{1} (R) : {inf}_{x \in R} f^{'} (x) > 0, {sup}_{x \in R} f^{'} (x) < 2}$ , and $F = {f \in C^{1} (R) : {inf}_{x \in R} f^{'} (x) > 0}$ . In the high dimensional cases, $R$ and $F$ are hard to illustrate. Nevertheless, the Lipschitz constants of the functions in $R$ is less than $2$ (behrmann2019invertible), but those of the functions in $F$ can be arbitrarily large. Based on Lemma 1, we prove that the function family of ImpFlows $I$ consists of the compositions of two functions in $F$ , and therefore is a strictly larger than $R_{2}$ , as summarized in the following theorem.

Theorem 2.

(Equivalent form of the function family of a single-block ImpFlow).

I=F2\coloneqq{f:f=f2∘f1 for some f1,f2∈F}.

(8)

Note that the identity mapping $I d \in F$ , and it is easy to get $F \subset I$ . Thus, the Lipschitz constant of a single ImpFlow (and its reverse) can be arbitrarily large. Because $R ⫋ F$ and there exists some functions in $I ∖ R_{2}$ (see a constructed example in Sec. 4.3), we can get the following corollary.

Corollary 1.

$R ⫋ R_{2} ⫋ F_{2} = I$ .

The results on the 1D example in Fig. 2 (b) and (c) accord with Corollary 1. Besides, Corollary 1 can be generalized to the cases with $2 ℓ$ -block ResFlows and $ℓ$ -block ImpFlows, which strongly motivates the usage of implicit layers in normalizing flows.

4.3 Comparison with multi-block ResFlows

We further investigate the relationship between $R_{ℓ}$ for $ℓ > 2$ and $I$ , as illustrated in Fig. 1 (b). For a fixed $ℓ$ , the Lipschitz constant of functions in $R_{ℓ}$ is still bounded, and there exist infinite functions that are not in $R_{ℓ}$ but in $I$ . We construct one such function family: for any $L, r \in R^{+}$ , define

P (L, r) = {f : f \in F, \exists B_{r} \subset R^{d}, \forall x, y \in B_{r}, ∥ f (x) - f (y) ∥_{2} \geq L ∥ x - y ∥_{2}},

(9)

where $B_{r}$ is an $d$ -dimensional ball with radius of $r$ . Obviously, $P (L, r)$ is an infinite set. Below, we will show that $\forall 0 < ℓ < {log}_{2} (L)$ , $R_{ℓ}$ has a non-negligible approximation error for functions in $P (L, r)$ . However, they are exactly representable by functions in $I$ .

Theorem 3.

Given $L > 0$ and $r > 0$ , we have

$P (L, r) \subset I$ .
$\forall 0 < ℓ < {log}_{2} (L)$ , $P (L, r) \cap R_{ℓ} = \emptyset$ . Moreover, for any $f \in P (L, r)$ with $d$ -dimensional ball $B_{r}$ , the minimal error for fitting $f$ in $B_{r}$ by functions in $R_{ℓ}$ satisfies

$inf g \in R_{ℓ} sup x \in B_{r} ∥ f (x) - g (x) ∥_{2} \geq \frac{r}{2} (L - 2^{ℓ})$ (10)

It follows Theorem 3 that to model $f \in P (L, r)$ , we need only a single-block ImpFlow but at least a ${log}_{2} (L)$ -block ResFlow. In Fig. 2 (b), we show a 1D case where a 3-block ResFlow cannot fit a function that is exactly representable by a single-block ImpFlow. In addition, we also prove some other properties of ImpFlows. In particular, $R_{3} \subset̸ I$ . We formally present the results in Appendix B.

5 Generative Modeling with ImpFlows

ImpFlows can be parameterized by neural networks and stacked to form a deep generative model to model high-dimensional data distributions. We develop a scalable algorithm to perform inference, sampling and learning in such models. For simplicity, we focus on a single-block during derivation.

Formally, a parametric ImpFlow block $z = f (x; θ)$ is defined by

(11)

and $L i p (g_{x}) < 1$ , $L i p (g_{z}) < 1$ . Let $θ$ denote all the parameters in $g_{x}$ and $g_{z}$ (which does NOT mean $g_{x}$ and $g_{z}$ share parameters). Note that $x$ refers to the input of the layer, not the input data.

The inference process to compute $z$ given $x$ in a single ImpFlow block is solved by finding the root of $F (z, x; θ) = 0$ w.r.t. $z$ , which cannot be explicitly computed because of the implicit formulation. Instead, we adopt a quasi-Newton method (i.e. Broyden’s method (broyden1965class)) to solve this problem iteratively, as follows:

z^{[i + 1]} = z^{[i]} - α B F (z^{[i]}, x; θ), for i = 0, 1, \dots,

(12)

where $B$ is a low-rank approximation of the Jacobian inverse¹¹1We refer readers to broyden1965class for the calculation details for $B$ . and $α$ is the step size which we use line search method to dynamically compute. The stop criterion is $∥ F (z^{[i]}, x; θ) ∥_{2} < ϵ_{f}$ , where $ϵ_{f}$

is a hyperparameter that balances the computation time and precision. As Theorem

1 guarantees the existence and uniqueness of the root, the convergence of the Broyden’s method is also guaranteed, which is typically faster than a linear rate.

Another inference problem is to estimate the log-likelihood. Assume that $z \sim p (z)$ where $p (z)$ is a simple prior distribution (e.g. standard Gaussian). The log-likelihood of $x$ can be written by

ln p (x) = ln p (z) + ln det (I + J_{g_{x}} (x)) - ln det (I + J_{g_{z}} (z)),

(13)

where $J_{f} (x)$ denotes the Jacobian matrix of a function $f$ at $x$ . See Appendix. A.4 for the detailed derivation. Exact calculation of the log-determinant term requires $O (d^{3})$

time cost and is hard to scale up to high-dimensional data. Instead, we propose the following unbiased estimator of

ln p (x)

using the same technique in chen2019residual with Skilling-Hutchinson trace estimator (skilling1989eigenvalues; hutchinson1989stochastic):

ln p (x) = ln p (z) + E_{n \sim p (N), v \sim N (0, I)} [n \sum k = 1 \frac{(- 1)^{k + 1}}{k} \frac{(v^{T} [J_{g_{x}} (x)^{k}] v - v^{T} [J_{g_{z}} (z)^{k}] v)}{P (N \geq k)}],

(14)

where $p (N)$ is a distribution supported over the positive integers.

The sampling process to compute $x$ given $z$ can also be solved by the Broyden’s method, and the hyperparameters are shared with the inference process.

In the learning

process, we perform stochastic gradient descent to minimize the negative log-likelihood of the data, denoted as

L

. For efficiency, we estimate the gradient w.r.t. the model parameters in the backpropagation manner. According to the chain rule and the additivity of the log-determinant, in each layer we need to estimate the gradients w.r.t.

x

and

θ

of Eqn. (13). In particular, the gradients computation involves two terms: one is

\pdv(⋅)lndet(I+Jg(x;θ))

and the other is

\pdvLz\pdvz(⋅)

, where

g

is a function satisfying

L i p (g) < 1

and

(\cdot)

denotes

x

θ

. On the one hand, for the log-determinant term, we can use the same technique as chen2019residual, and obtain an unbiased gradient estimator as follows.

\pdvlndet(I+Jg(x;θ))(⋅)=En∼p(N),v∼N(0,I)[(n∑k=0(−1)kP(N≥k)vTJg(x;θ)k)\pdvJg(x;θ)(⋅)v],

(15)

where $p (N)$ is a distribution supported over the positive integers. On the other hand, $\pdvLz\pdvz(⋅)$ can be computed according to the implicit function theorem as follows (See details in Appendix A.5):

\pdvLz\pdvz(⋅)=\pdvLzJ−1G(z)\pdvF(z,x;θ)(⋅), where G(z;θ)=gz(z;θ)+z.

(16)

In comparision to directly calculate the gradient through the quasi-Newton iterations of the forward pass, the implicit gradient above is simple and memory-efficient, treating the root solvers as a black-box. Following bai2019deep, we compute $\pdvLzJ−1G(z)$ by solving a linear system iteratively, as detailed in Appendix C.1. The training algorithm is formally presented in Appendix C.4.

6 Experiments

We demonstrate the model capacity of ImpFlows on the classification and density modeling tasks²²2 See https://github.com/thu-ml/implicit-normalizing-flows for details.. In all experiments, we use spectral normalization (miyato2018spectral) to enforce the Lipschitz constrants, where the Lipschitz constant upper bound of each layer (called Lipschitz coefficient) is denoted as $c$ . For the Broyden’s method, we use $ϵ_{f} = 10^{- 6}$ and $ϵ_{b} = 10^{- 10}$ for training and testing to numerically ensure the invertibility and the stability during training. Please see other detailed settings including the method of estimating the log-determinant, the network architecture, learning rate, batch size, and so on in Appendix D.

6.1 Verifying Capacity on Classification

		Vanilla	$c = 0.99$	$c = 0.9$	$c = 0.8$	$c = 0.7$	$c = 0.6$
CIFAR10	ResFlow	6.61( $\pm$ 0.02)	8.24	8.39	8.69	9.25	9.94
	ResFlow		( $\pm$ 0.03)	( $\pm$ 0.01)	( $\pm$ 0.03)	( $\pm$ 0.02)	( $\pm$ 0.02)
	ImpFlow		7.29	7.41	7.94	8.44	9.22
	ImpFlow		( $\pm$ 0.03)	( $\pm$ 0.03)	( $\pm$ 0.06)	( $\pm$ 0.04)	( $\pm$ 0.02)
CIFAR100	ResFlow	27.83( $\pm$ 0.03)	31.02	31.88	32.21	33.58	34.48
	ResFlow		( $\pm$ 0.05)	( $\pm$ 0.02)	( $\pm$ 0.03)	( $\pm$ 0.02)	( $\pm$ 0.03)
	ImpFlow		29.06	30.47	31.40	32.64	34.17
	ImpFlow		( $\pm$ 0.03)	( $\pm$ 0.03)	( $\pm$ 0.03)	( $\pm$ 0.01)	( $\pm$ 0.02)

Table 1: Classification error rate (%) on test set of vanilla ResNet, ResFlow and ImpFlow of ResNet-18 architecture, with varying Lipschitz coefficients

c

	POWER	GAS	HEPMASS	MINIBOONE	BSDS300
RealNVP (realnvp)	0.17	8.33	-18.71	-13.55	153.28
FFJORD (grathwohl2018ffjord)	0.46	8.59	-14.92	-10.43	157.40
MAF (papamakarios2017masked)	0.24	10.08	-17.70	-11.75	155.69
NAF (huang2018neural)	0.62	11.96	-15.09	-8.86	157.73
ImpFlow ( $L = 20$ )	0.61	12.11	-13.95	-13.32	155.68
ResFlow ( $L = 10$ )	0.26	6.20	-18.91	-21.81	104.63
ImpFlow ( $L = 5$ )	0.30	6.94	-18.52	-21.50	113.72

Table 2: Average test log-likelihood (in nats) of tabular datasets. Higher is better.

	$c = 0.9$	$c = 0.8$	$c = 0.7$	$c = 0.6$
ResFlow ( $L = 12$ )	3.469( $\pm$ 0.0004)	3.533( $\pm$ 0.0002)	3.627( $\pm$ 0.0004)	3.820( $\pm$ 0.0003)
ImpFlow ( $L = 6$ )	3.452( $\pm$ 0.0003)	3.511( $\pm$ 0.0002)	3.607( $\pm$ 0.0003)	3.814( $\pm$ 0.0005)

Table 3: Average bits per dimension of ResFlow and ImpFlow on CIFAR10, with varying Lipschitz coefficients

c

. Lower is better.

We first empirically compare ResFlows and ImpFlows on classification tasks. Compared with generative modeling, classification is a more direct measure of the richness of the functional family, because it isolates the function fitting from generative modeling subtleties, such as log-determinant estimation. We train both models in the same settings on CIFAR10 and CIFAR100 (krizhevsky2009learning). Specifically, we use an architecture similar to ResNet-18 (he2016deep). Overall, the amount of parameters of ResNet-18 with vanilla ResBlocks, ResFlows and ImpFlows are the same of $6.5$ M. The detailed network structure can be found in Appendix D. The classification results are shown in Table 1. To see the impact of the Lipschitz constraints, we vary the Lipschitz coefficient $c$ to show the difference between ResFlows and ImpFlows under the condition of a fixed Lipschitz upper bound. Given different values of $c$ , the classification results of ImpFlows are consistently better than those of ResFlows. These results empirically validate Corollary 1, which claims that the functional family of ImpFlows is richer than ResFlows. Besides, for a large Lipschitz constant upper bound $c$ , ImpFlow blocks are comparable with the vanilla ResBlocks in terms of classification.

6.2 Density Modeling On 2D Toy Data

Figure 3: Checkerboard data density and the results of a 8-block ResFlow and a 4-block ImpFlow.

For the density modeling tasks, we first evaluate ImpFlows on the Checkerboard data whose density is multi-modal, as shown in Fig. 3 (a). For fairness, we follow the same experiment settings as chen2019residual (which are specified in Appendix D), except that we adopt a Sine (sitzmann2020implicit)activation function for all models. We note that the data distribution has a bounded support while we want to fit a transformation $f$

mapping it to the standard Gaussian distribution, whose support is unbounded. A perfect

f

requires a sufficiently large

∥ J_{f} (x) ∥_{2}

for some

x

mapped far from the mean of the Gaussian. Therefore, the Lipschtiz constant of such

f

is too large to be fitted by a ResFlow with 8 blocks (See Fig. 3 (b)). A

4

-block ImpFlow can achieve a result of

5.05

bits, which outperforms the

5.08

bits of a

8

-block ResFlow with the same number of parameters. Such results accord with our theoretical results in Theorem 2 and strongly motivate ImpFlows.

6.3 Density Modeling On Real Data

We also train ImpFlows on some real density modeling datasets, including the tabular datasets (used by papamakarios2017masked), CIFAR10 and 5-bit $64 \times 64$ CelebA (kingma2018glow). For all the real datasets, we use the scalable algorithm proposed in Sec. 5.

We test our performance on five tabular datasets: POWER ( $d = 6$ ), GAS ( $d = 8$ ), HEPMASS ( $d = 21$ ), MINIBOONE ( $d = 43$ ) and BSDS300 ( $d = 63$ ) from the UCI repository (Dua:2019), where $d$ is the data dimension. For a fair comparison, on each dataset we use a 10-block ResFlow and a 5-block ImpFlow with the same amount of parameters, and a 20-block ImpFlow for a better result. The detailed network architecture and hyperparameters can be found in Appendix D. Table 2 shows the average test log-likelihood for ResFlows and ImpFlows. ImpFlows achieves better density estimation performance than ResFlow consistently on all datasets. Again, the results demonstrate the effectiveness of ImpFlows.

Then we test ImpFlows on the CIFAR10 dataset. We train a multi-scale convolutional version for both ImpFlows and ResFlows, following the same settings as chen2019residual except that we use a smaller network of 5.5M parameters for both ImpFlows and ResFlows (see details in Appendix D). As shown in Table 3, Impflow achieves better results than ResFlow consistently given different values of the Lipschitz coefficient $c$ . Moreover, the computation time of ImpFlow is comparable to that of ResFlow. See Appendix C.2

for detailed results. Besides, there is a trade-off between the expressiveness and the numerical optimization of ImpFlows in larger models. Based on the above experiments, we believe that advances including an lower-variance estimate of the log-determinant can benefit ImpFlows in larger models, which is left for future work.

We also train ImpFlows on the 5-bit $64 \times 64$ CelebA. For a fair comparison, we use the same settings as chen2019residual. The samples from our model are shown in Appendix E.

7 Conclusions

We propose implicit normalizing flows (ImpFlows), which generalize normalizing flows via utilizing an implicit invertible mapping defined by the roots of the equation $F (z, x) = 0$ . ImpFlows build on Residual Flows (ResFlows) with a good balance between tractability and expressiveness. We show that the functional family of ImpFlows is richer than that of ResFlows, particularly for modeling functions with large Lipschitz constants. Based on the implicit differentiation formula, we present a scalable algorithm to train and evaluate ImpFlows. Empirically, ImpFlows outperform ResFlows on several classification and density modeling benchmarks. Finally, while this paper mostly focuses on the implicit generalization of ResFlows, the general idea of utilizing implicit functions for NFs could be extended to a wider scope. We leave it as a future work.

Acknowledgement

We thank Yuhao Zhou, Shuyu Cheng, Jiaming Li, Kun Xu, Fan Bao, Shihong Song and Qi’An Fu for proofreading. This work was supported by the National Key Research and Development Program of China (Nos. 2020AAA0104304), NSFC Projects (Nos. 61620106010, 62061136001, U19B2034, U181146, 62076145), Beijing NSF Project (No. JQ19016), Beijing Academy of Artificial Intelligence (BAAI), Tsinghua-Huawei Joint Research Program, Huawei Hisilicon Kirin Intelligence Engineering Development, the MindSpore team, a grant from Tsinghua Institute for Guo Qiang, Tiangong Institute for Intelligent Computing, and the NVIDIA NVAIL Program with GPU/DGX Acceleration. C. Li was supported by the fellowship of China postdoctoral Science Foundation (2020M680572), and the fellowship of China national postdoctoral program for innovative talents (BX20190172) and Shuimu Tsinghua Scholar. J. Chen was supported by Shuimu Tsinghua Scholar.

References

Appendix A Additional Lemmas and Proofs

a.1 Proof For Theorem 1

Proof.

(Theorem 1)

Firstly, $\forall x_{0} \in R^{d}$ , the mapping

is a contrative mapping, which can be shown by Lipschitz condition of $g_{z}$ :

∥ (F (z_{1}, x_{0}) + z_{1}) - (F (z_{2}, x_{0}) + z_{2}) ∥ = ∥ g_{z} (z_{1}) - g_{z} (z_{2}) ∥ < ∥ z_{1} - z_{2} ∥ .

Therefore, $h_{x_{0}} (z)$ has an unique fixed point, denoted by $f (x_{0})$ :

h_{x_{0}} (f (x_{0})) = f (x_{0}) \Leftrightarrow F (f (x_{0}), x_{0}) = 0

Similarly, we also have: $\forall z_{0} \in R^{d}$ , there exists an unique $g (z_{0})$ satisfying $F (z_{0}, g (z_{0})) = 0$ .

Moreover, Let $z_{0} = f (x_{0})$ , we have $F (f (x_{0}), g (f (x_{0}))) = 0$ . By the uniqueness, we have $g (f (x_{0})) = x_{0}, \forall x_{0} \in R^{d}$ . Similarly, $f (g (x_{0})) = x_{0}, \forall x_{0} \in R^{d}$ . Therefore, $f$ is unique and invertible. ∎

a.2 Proof For Theorem 2

We denote $D$ as the set of all bi-Lipschitz $C^{1}$ -diffeomorphisms from $R^{d}$ to $R^{d}$ .

Firstly, we prove Lemma 1 in the main text.

Proof.

(Lemma 1). $\forall f \in R$ , we have

sup x \in R^{d} ∥ J_{f} (x) - I ∥_{2}^{2} < 1,

which is equivalent to

	$sup x \in R^{d}, v \in R^{d}, ∥ v ∥_{2} = 1 ∥ (J_{f} (x) - I) v ∥_{2}^{2} < 1 (% Definition of operator norm.)$
	$sup x \in R^{d}, v \in R^{d}, ∥ v ∥_{2} = 1 v^{T} (J_{f}^{T} (x) - I) (J_{f} (x) - I) v < 1$

Note that $J_{f} (x)$ is nonsingular, so $\forall x, v \in R^{d}, ∥ v ∥_{2} = 1$ , we have $v^{T} J_{f}^{T} (x) J_{f} (x) v > 0$ . Thus,

So we have

inf x \in R^{d}, v \in R^{d}, ∥ v ∥_{2} = 1 v^{T} J_{f} (x) v > 0.

Note that the converse is not true, because $v^{T} J_{f} (x) v > 0$ does not restrict the upper bound of Lipschitz constant of $f$ . For example, when $f (x) = m x$ where $m$ is a positive real number, we have

inf x \in R^{d}, v \in R^{d}, ∥ v ∥_{2} = 1 v^{T} J_{f} (x) v = inf x \in R^{d}, v \in R^{d}, ∥ v ∥_{2} = 1 v^{T} (m I) v = m > 0

However, $m$ can be any large positive number. So we have $R ⫋ F$ . ∎

Lemma 2.

$\forall f \in D$ , if

inf \begin{matrix} x \in R^{d}, v \in R^{d}, ∥ v ∥_{2} = 1 \end{matrix} v^{T} J_{f} (x) v > 0,

(17)

then

inf \begin{matrix} x \in R^{d}, v \in R^{d}, ∥ v ∥_{2} = 1 \end{matrix} v^{T} J_{f^{- 1}} (x) v > 0,

(18)

Proof.

(Proof of Lemma 2). By Inverse Function Theorem,

J_{f^{- 1}} (x) = J_{f}^{- 1} (f^{- 1} (x)) .

Because $f$ is from $R^{d}$ to $R^{d}$ , we have

	$inf x \in R^{d}, v \in R^{d}, ∥ v ∥_{2} = 1 v^{T} J_{f^{- 1}} (x) v$	$= inf x \in R^{d}, v \in R^{d}, ∥ v ∥_{2} = 1 v^{T} J_{f}^{- 1} (f^{- 1} (x)) v$
		$= inf x \in R^{d}, v \in R^{d}, ∥ v ∥_{2} = 1 v^{T} J_{f}^{- 1} (x) v$

Let and $v_{0} = \frac{u}{∥ u ∥_{2}}$ , we have $∥ v_{0} ∥_{2} = 1$ , and

v^{T} J_{f}^{- 1} (x) v = u^{T} J_{f}^{T} (x) u = u^{T} J_{f} (x) u = ∥ u ∥_{2}^{2} v_{0}^{T} J_{f} (x) v_{0} .

The above equation uses this fact: for a real $d \times d$ matrix $A$ , $\forall x \in R^{d}, x^{T} A x = (x^{T} A x)^{T} = x^{T} A^{T} x$ because $x^{T} A x \in R$ .

Note that $f$ is Lipschitz continuous, $∥ J_{f} (x) ∥_{2} \leq L i p (f)$ . So

which means

∥ u ∥_{2} \geq \frac{1}{L i p (f)} .

Thus,

	$inf x \in R^{d}, v \in R^{d}, ∥ v ∥_{2} = 1 v^{T} J_{f}^{- 1} (x) v$	$= inf x \in R^{d}, v \in R^{d}, ∥ v ∥_{2} = 1 ∥ u ∥_{2}^{2} v_{0}^{T} J_{f} (x) v_{0}$
		$\geq inf x \in R^{d}, u \in R^{d}, ∥ J_{f} (x) u ∥_{2} = 1 ∥ u ∥_{2}^{2} inf x \in R^{d}, v_{0} \in R^{d}, ∥ v_{0} ∥_{2} = 1 v_{0}^{T} J_{f} (x) v_{0}$
		$\geq \frac{1}{L i p (f)^{2}} inf x \in R^{d}, v \in R^{d}, ∥ v ∥_{2} = 1 v^{T} J_{f} (x) v$
		$> 0$

∎

Lemma 3.

$\forall f \in D$ , if $f^{- 1} \in R$ , we have

inf \begin{matrix} x \in R^{d}, v \in R^{d}, ∥ v ∥_{2} = 1 \end{matrix} v^{T} J_{f} (x) v > 0.

(19)

Proof.

(Proof of Lemma 3). $\forall f \in D$ , if $f^{- 1} \in R$ , then from Lemma 1, we have

inf x \in R^{d}, v \in R^{d}, ∥ v ∥_{2} = 1 v^{T} J_{f^{- 1}} (x) v > 0.

Note that $f^{- 1} \in D$ , from Lemma 2 we have

inf x \in R^{d}, v \in R^{d}, ∥ v ∥_{2} = 1 v^{T} J_{f} (x) v > 0.

∎

Lemma 4.

$\forall f \in D$ , if

inf x \in R^{d}, v \in R^{d}, ∥ v ∥_{2} = 1 v^{T} J_{f} (x) v > 0,

then $\exists α_{0} > 0$ , s.t. $\forall 0 < α < α_{0}$ ,

sup x \in R^{d} ∥ α J_{f} (x) - I ∥_{2} < 1.

Proof.

(Proof of Lemma 4). Note that $f$ is Lipschitz continuous, so $L i p (f) = {sup}_{x \in R^{d}} ∥ J_{f} (x) ∥_{2}$ . Denote

β = inf x \in R^{d}, v \in R^{d}, ∥ v ∥_{2} = 1 v^{T} J_{f} (x) v .

And let

α_{0} = \frac{β}{L i p (f)^{2}} > 0.

$\forall 0 < α < α_{0}$ , we have

	$sup x \in R^{d} ∥ α J_{f} (x) - I ∥_{2}^{2}$	$= sup x \in R^{d}, v \in R^{d}, ∥ v ∥_{2} = 1 v^{T} (α J_{f}^{T} (x) - I) (α J_{f} (x) - I) v$
		$= 1 + sup x \in R^{d}, v \in R^{d}, ∥ v ∥_{2} = 1 α^{2} v^{T} J_{f}^{T} (x) J_{f} (x) v - 2 α v^{T} J_{f} (x) v$
		$\leq 1 + α^{2} sup x \in R^{d}, v \in R^{d}, ∥ v ∥_{2} = 1 v^{T} J_{f}^{T} (x) J_{f} (x) v$
		$+ 2 α sup x \in R^{d}, v \in R^{d}, ∥ v ∥_{2} = 1 (- v^{T} J_{f} (x) v)$
		$= 1 + α^{2} sup x \in R^{d}, v \in R^{d}, ∥ v ∥_{2} = 1 v^{T} J_{f}^{T} (x) J_{f} (x) v$
		$- 2 α inf x \in R^{d}, v \in R^{d}, ∥ v ∥_{2} = 1 v^{T} J_{f} (x) v$
		$= 1 + α^{2} sup x \in R^{d} ∥ J_{f} (x) ∥_{2}^{2} - 2 α β$
		$= 1 + α (α L i p (f)^{2} - 2 β)$
		$< 1 + α (α_{0} L i p (f)^{2} - 2 β)$
		$= 1 - α β$
		$< 1.$

The above equation uses this fact: for a real $d \times d$ matrix $A$ , $\forall x \in R^{d}, x^{T} A x = (x^{T} A x)^{T} = x^{T} A^{T} x$ because $x^{T} A x \in R$ . ∎

Proof.

(Theorem 2) Denote

\begin{matrix} P = {f \in D | & \exists f_{1}, f_{2} \in D, f = f_{2} \circ f_{1}, where inf x \in R^{d}, v \in R^{d}, ∥ v ∥_{2} = 1 v^{T} J_{f_{1}} (x) v > 0, inf x \in R^{d}, v \in R^{d}, ∥ v ∥_{2} = 1 v^{T} J_{f_{2}} (x) v > 0} . \end{matrix}

Firstly, we show that $I \subset P$ . $\forall f \in I$ , assume $f = f_{2} \circ f_{1}$ , where $f_{1} \in R$ and $f_{2}^{- 1} \in R$ . By Lemma 1 and Lemma 3, we have

	$inf x \in R^{d}, v \in R^{d}, ∥ v ∥_{2} = 1 v^{T} J_{f_{1}} (x) v$	$> 0,$
	$inf x \in R^{d}, v \in R^{d}, ∥ v ∥_{2} = 1 v^{T} J_{f_{2}} (x) v$	$> 0.$

Thus, $f \in P$ . So $I \subset P$ .

Next, we show that $P \subset I$ . $\forall f \in P$ , assume $f = f_{2} \circ f_{1}$ , where

	$inf x \in R^{d}, v \in R^{d}, ∥ v ∥_{2} = 1 v^{T} J_{f_{1}} (x) v$	$> 0,$
	$inf x \in R^{d}, v \in R^{d}, ∥ v ∥_{2} = 1 v^{T} J_{f_{2}} (x) v$	$> 0.$

From Lemma 2, we have

inf x \in R^{d}, v \in R^{d}, ∥ v ∥_{2} = 1 v^{T} J_{f_{2}^{- 1}} (x) v

> 0.

From Lemma 4, $\exists α_{1} > 0, α_{2} > 0$ , s.t. $\forall 0 < α < min {α_{1}, α_{2}}$ ,

	$sup x \in R^{d} ∥ α J_{f_{1}} (x) - I ∥_{2} < 1,$
	$sup x \in R^{d} ∥ α J_{f_{2}^{- 1}} (x) - I ∥_{2} < 1.$

Let $α = \frac{1}{2} min {α_{1}, α_{2}}$ . Let $g = g_{2} \circ g_{1}$ , where

	$g_{1} (x)$	$= α f_{1} (x),$
	$g_{2} (x)$	$= f_{2} (\frac{x}{α}) .$

We have $g (x) = f_{2} (\frac{α f_{1} (x)}{α}) = f (x)$ , and

	$J_{g_{1}} (x)$	$= α J_{f_{1}} (x),$
	$g_{2}^{- 1} (x)$	$= α f_{2}^{- 1} (x),$
	$J_{g_{2}^{- 1}} (x)$	$= α J_{f_{2}^{- 1}} (x) .$

So we have

	$sup x \in R^{d} ∥ J_{g_{1}} (x) - I ∥_{2} = sup x \in R^{d} ∥ α J_{f_{1}} (x) - I ∥_{2} < 1,$
	$sup x \in R^{d} ∥ J_{g_{2}^{- 1}} (x) - I ∥_{2} = sup x \in R^{d} ∥ α J_{f_{2}^{- 1}} (x$

Implicit Normalizing Flows

Authors

Equivariant Flows: sampling configurations for multi-body systems with symmetric energies

Universal Approximation of Residual Flows in Maximum Mean Discrepancy

Quasi-Autoregressive Residual (QuAR) Flows

The Convolution Exponential and Generalized Sylvester Flows

Gravitational-wave parameter estimation with autoregressive neural network flows

Jacobian Determinant of Normalizing Flows

Parallel multilevel restricted Schwarz preconditioners for implicit simulation of subsurface flows with Peng-Robinson equation of state

Code Repositories