Diffusion models have emerged as a promising class of generative models that can generate high quality images Song and Ermon (2019); Song et al. (2020); Nichol and Dhariwal (2021), outperforming GANs on perceptual quality metrics Dhariwal and Nichol (2021)
, and likelihood-based models on density estimationKingma et al. (2021). One of the limitations of these models, however, is the fact that they require a long diffusion chain (many repeated applications of a denoising process), in order to generate high-fidelity samples. Several recent papers have focused on tackling this limitation, e.g., by shortening the length of diffusion process through an alternative parameterization Song et al. (2020); Kong and Ping (2021), or through progressive distillation of a sampler with large diffusion chain into a smaller one Luhman and Luhman (2021); Salimans and Ho (2022). However, all of these methods still rely on a fundamentally sequential sampling process, imposing challenges on accelerating the sampling and for other applications like differentiating through the entire generation process.
In this paper, we propose an alternative approach that also begins to address such challenges from a different perspective. Specifically, we propose to model the generative process of a specific class of diffusion model, the denoising diffusion implicit model (DDIM) Song et al. (2020), as a deep equilibrium (DEQ) model Bai et al. (2019). Deep equilibrium (DEQ) models are networks that aim to find the fixed point of the underlying system in the forward pass and differentiate implicitly through this fixed point in the backward pass. To apply DEQs to diffusion models, we first formulate this process as an equilibrium system consisting of all joint sampling steps, and then simultaneously solve the fixed point of all the steps to achieve sampling.
This approach has several benefits: First, the DEQ sampling process can be solved in parallel over multiple GPUs by batching the workload. This is particularly beneficial in the case of single image (i.e., batch-size-one) generation, where the serial sampling nature of diffusion models has inevitably made them unable to maximize GPU computation. Second, solving for the joint equilibria simultaneously leads to faster overall convergence as we have better estimates of the intermediate states in fewer steps. Specifically, the formulation naturally lends itself to an augmented diffusion chain that each state is generated according to all others. Third, the DEQ formulation allows us to leverage a faster differentiation through the chain. This enables us to much more effectively solve problems that require us to differentiate through the generative process, useful for tasks such as model inversion that seeks to find the noise that leads to a particular instance of an image.
We demonstrate the advantages of this formulation on two applications: single image generation and model inversion. Both single image generation and model inversion are widely applied in real world image manipulation tasks like image editing and restoration (Zhu et al., 2020; Saeedi et al., 2018; Issenhuth et al., 2021; Abdal et al., 2021; Ling et al., 2021; Meng et al., 2021)
. On CIFAR-10Krizhevsky (2009) and CelebA Liu et al. (2015), DEQ achieves up to 2 speedup over the sequential sampling processes of DDIM Song et al. (2020), while maintaining a comparable perceptual quality of images. In the model inversion, the loss converges much faster when trained with DEQs than with sequential sampling. Moreover, optimizing the sequential sampling process can be computationally expensive. Leveraging modern autograd packages is infeasible as they require storing the entire computational graph for all states. Some recent works like Nie et al. (2022) achieve this through use of SDE solvers. In contrast, with DEQs we can use the implicit differentiation of memory complexity. Empirically, the initial hidden state recovered by DEQ more accurately regenerates the original image while capturing its finer details.
To summarize, our main contributions are as follows:
We formulate the generative process of an augmented type of DDIM as a deep equilibrium model that allows the use of black-box solvers to efficiently compute the fixed point and generate images.
The DEQ formulation parallelizes the sampling process of DDIM, and as a result, it can be run on multiple GPUs instead of a single GPU. This alternate sampling process converges faster compared to the original process.
We demonstrate the advantages of our formulation on single image generation and model inversion. We find that optimizing the sampling process via DEQs consistently outperforms naive sequential sampling.
Denoising diffusion probabilistic models (DDPM) (Sohl-Dickstein et al., 2015; Ho et al., 2020) are generative models that can convert the data distribution to a simple distribution, (e.g., a standard Gaussian, ), through a diffusion process. Specifically, given samples from a target distribution
, the diffusion process is a Markov chain that adds Gaussian noises to the data to generate latent statesin the same sample space as . The inference distribution of diffusion process is given by:
To learn the parameters that characterize a distribution as an approximation of , a surrogate variational lower bound (Sohl-Dickstein et al., 2015) was proposed to train this model:
After training, samples can be generated by a reverse Markov chain, i.e., first sampling , and then repeatedly sampling till we reach .
As noted in (Sohl-Dickstein et al., 2015; Song et al., 2020), the length of a diffusion process is usually large (e.g., Ho et al. (2020)) as it contributes to a better approximation of Gaussian conditional distributions in the generative process. However, because of the large value of , sampling from diffusion models can be visibly slower compared to other deep generative models like GANs (Goodfellow et al., 2014).
One feasible acceleration is to rewrite the forward process into a non-Markovian one that leads to a “shorter” and deterministic generative process, i.e., denoising diffusion implicit model (Song et al., 2020) (DDIM). DDIM can be trained similarly to DDPM, using the variational lower bound shown in Eq. 2. Essentially, DDIM constructs a nearly non-stochastic scheme that can quickly sample from the learned data distribution without introducing additional noises. Specifically, the scheme to generate a sample given is:
where , , and is an estimator trained to predict the noise given a noisy state
. For a variance schedule, we use the notation . Different values of define different generative processes. When for all , the generative process represents a DDPM. Setting for all gives rise to a DDIM, which results in a deterministic generating process except the initial sampling .
Deep Equilibrium Models
Deep equilibrium models are a recently-proposed class of deep networks that, in their forward pass, seek to find a fixed point of a single layer applied repeatedly to a hidden state. Specifically, consider a deep feedforward model with layers:
where is the input injection, is the hidden state of layer, and is a layer that defines the feature transformation. Assuming the above model is weight-tied, i.e., , then in the limit of infinite depth, the output of this network converges to a fixed point .
Inspired from the neural convergence phenomenon, Deep equilibrium (DEQ) models (Bai et al., 2019) are proposed to directly compute this fixed point as the output, i.e.,
The equilibrium state can be solved by black-box solvers like Broyden’s method (Broyden, 1965), or Anderson acceleration (Anderson, 1965). To train this fixed-point system, Bai et al. (2019) leverage implicit differentiation to directly backpropagate through the equilibrium state using
memory complexity. DEQ is known as a principled framework for characterizing convergence and energy minimization in deep learning. We leave a detailed discussion inSec. 6.
3 A Deep Equilibrium Approach to DDIMs
In this section, we present the main modeling contribution of the paper, a formulation of diffusion processes under the DEQ framework. Although diffusion models may seem to be a natural fit for DEQ modeling (after all, we typically do not care about intermediate states in the denoising chain, but only the final clean image), there are several reasons why setting up the diffusion chain “naively” as a DEQ (i.e., making be a single sampling step) does not ultimately lead to a functional algorithm. Most fundamentally, the diffusion process is not time-invariant (i.e., not “weight-tied” in the DEQ sense), and the final generated image is practically-speaking independent of the noise used to generate it (i.e., not truly based upon “input injection” either).
Thus, at a high level, our approach to building a DEQ version of the DDIM involves representing all the states simultaneously within the DEQ state. The advantage of this approach is that 1) we can exactly capture the typical diffusion inference chain; and 2) we can create a more expressive reverse process where the state is updated based upon all previous states , improving the inference process; 3) we can execute all steps of the inference chain in parallel rather than solely in sequence as is typically required in diffusion models; and 4) we can use common DEQ acceleration methods, such as the Anderson solver Anderson (1965) to find the fixed point, which makes the sampling process converge faster. A downside of this formulation is that we need to store all DEQ states simultaneously (i.e., only the images, not the intermediate network states).
3.1 A DEQ formulation of DDIMs (DEQ-DDIM)
The generative process of DDIM is given by:
This process also lets us generate a sample using a subset of latent states , where . While this helps in accelerating the overall generative process, there is a tradeoff between sampling quality and computational efficiency. As noted in Song et al. (2020), larger values lead to lower FID scores of the generated images but need more compute time; smaller are faster to sample from, but the resulting images have worse FID scores.
Reformulating this sampling process as a DEQ addresses multiple concerns raised above. We can define a DEQ, with a sequence of latent states as its internal state, that simultaneously solves for the equilibrium points at all the timesteps. The global convergence of this process is upper bounded by steps, by definition. To derive the DEQ formulation of the generative process, first we rearrange the terms in Eq. (7):
Let . Then we can write
By induction, we can rewrite the above equation as:
This defines a “fully-lower-triangular” inference process, where the update of depends on the noise prediction network applied to all subsequent states ; in contrast to the traditional diffusion process, which updates based only on . Specifically, let represent the function that performs the operations in the equations (10) for a latent at timestep , and let represent the function that performs the same set of operations across all the timesteps simultaneously. We can write the above set of equations as a fixed point system:
The above system of equations represent a DEQ with as input injection. We can simultaneously solve for the roots of this system of equations through black-box solvers like Anderson acceleration (Anderson, 1965). Let , then we have
This DEQ formulation has multiple benefits. Solving for all the equilibria simultaneously leads to a better estimation of the intermediate latent states in a fewer number of steps (i.e., steps for ). This leads to faster convergence of the sampling process as the final sample , which is dependent on the latent states of all the previous time steps, has a better estimate of these intermediate latent states. Note that by the same reasoning, the intermediate latent states converge faster too. Thus, we can get images with perceptual quality comparable to DDIM in a significantly fewer number of steps. Of course, we also note that the computational requirements of each individual step has significantly increased, but this is at least largely offset by the fact that the steps can be executed as mini-batched in parallel over each state. Empirically, in fact, we often notice significant speedup using this approach on tasks like single image generation.
This DEQ formulation of DDIM can be extended to the stochastic generative processes of DDIM with , including that of DDPM (referred to as DEQ-sDDIM). The key idea is to sample noises for all the time steps along the sampling chain and treat this noise as an input injection to DEQ, in addition to .
where RootSolver() is any black-box fixed point solver, and represents the input injected noises. We discuss this formulation in more detail in Appendix D.
4 Efficient Inversion of DDIM
One of the primary strengths of DEQs is their constant memory consumption, for both forward pass and backward pass, regardless of their ‘effective depth’. This leads to an interesting application of DEQs in inverting DDIMs that fully leverages this advantage along with the other benefits discussed in the previous section.
the total number of epochs
4.1 Problem Setup
Given an arbitrary image , and a denoising diffusion model trained on a dataset , model inversion seeks to determine the latent that can generate an image identical to the original image through the generative process for DDIM described in Eq. (7). For an input image , and a generated image , this task needs to minimize the squared-Frobenius distance between these images:
4.2 Inverting DDIM: The Naive Approach
A relatively straightforward way to invert DDIM is to randomly sample , and update it via gradient descent by first estimating using the generative process in Eq. (7) and backpropagating through this process after computing the loss objective in (14). The overall process has been summarized in Algorithm 1. This process has a large computational overhead. Every training epoch requires a sequential sampling for all timesteps. Optimizing through this generative process would require the creation of a large computational graph for storing relevant intermediate variables necessary for the backward pass. Sequential sampling further slows down the entire process.
4.3 Efficient Inversion of DDIM with DEQs
Alternatively, we can use the DEQ formulation to develop a much more efficient inversion method. We provide a high-level overview of this approach in Algorithm 2. We can apply implicit function theorem (IFT) to the fixed point, i.e., (12) to compute gradients of the loss in (14) w.r.t. :
where could be any of the latent states , and is the inverse Jacobian of evaluated at . Refer to Bai et al. (2019) for a detailed proof. Computing the inverse of Jacobian matrix can become computationally intractable, especially when the latent states are high dimensional. Further, prior works (Bai et al., 2019, 2021; Geng et al., 2021b) have reported growing instability of DEQs during training due to the ill-conditioning of Jacobian. Recent works (Geng et al., 2021a; Fung et al., 2021; Geng et al., 2021b; Bai et al., 2022a) suggest that we do not need an exact gradient to train DEQs. We can instead use an approximation to Eq. (15), i.e.,
where is an approximation of . For example, (Geng et al., 2021a; Fung et al., 2021; Geng et al., 2021b) show that setting , i.e., 1-step gradient, works well. In this work, we follow Geng et al. (2021b) to further add a damping factor to the 1-step gradient. The forward pass is given by:
The gradients for the backward pass can be computed through standard autograd packages. We provide the PyTorch-style pseudocode of our approach in theAppendix B. Using inexact gradients for the backward pass has several benefits: 1) It remarkably improves the training stability of DEQs; 2) Our backward pass consists of a single step and is ultra-cheap to compute. It reduces the total training time by a significant amount. It is easy to extend the strategy used in Algorithm 2 and use DEQs to invert DDIMs with stochastic generative process (referred to as DEQ-sDDIM). We provide the key steps of this approach in Algorithm 4.
We consider four datasets that have images of different resolutions for our experiments: CIFAR10 (3232) Krizhevsky (2009), CelebA (6464) Liu et al. (2015), LSUN Bedroom (256256) and LSUN Outdoor Church (256256) Yu et al. (2015). For all the experiments, we use Anderson acceleration as the default fixed point solver. We use the pretrained denoising diffusion models from Ho et al. (2020) for CIFAR10, LSUN Bedroom, and LSUN Outdoor Church, and from Song et al. (2020) for CelebA. While training DEQs for model inversion, we use the 1-step gradient Eq. 18 to compute the backward pass. The damping factor for 1-step gradient is set to . All the experiments have been performed on NVIDIA RTX A6000 GPUs. We provide additional experimental details in the Appendix A. While the primary focus in this section will be on the DDIM with a deterministic generative process i.e., , we also include a few key results on stochastic version of DDIM (DEQ-sDDIM) here. More extensive experiments can be found in Appendix D.
5.1 Convergence of DEQ-DDIM
We verify that DEQ-DDIM converges to a fixed point by plotting the values of over Anderson solver steps. As seen in Figure 1, DEQ-DDIM converges to a fixed point for generative processes of different lengths. It is easier to reach simultaneous equilibria on smaller sequence lengths than larger sequence lengths. However, this does not affect the quality of images generated. We visualize the latent states of DEQ-DDIM in Figure 2. Our experiments demonstrate that DEQ-DDIM is able to generate high-quality images in as few as 15 Anderson solver steps on diffusion chains that were trained on a much larger number of steps . One might note that DEQ-DDIM converges to a limit cycle for diffusion processes with larger sequence lengths. This is not a limitation as we only want the latent states at the last few timesteps to converge well, which happens in practice as demonstrated in Fig. 2. Further, these residuals can be driven down by using more powerful solvers like quasi-Newton methods, e.g., Broyden’s method.
5.2 Sample quality of images generated with DEQ-DDIM
We verify that DEQ-DDIM can generate images of comparable quality to DDIM by reporting Fréchet Inception Distance (FID) Heusel et al. (2017) in Table 1. For the forward pass of DEQ-DDIM, we run Anderson solver for a maximum of 15 steps for each image. We report FID scores on 50,000 images, and average time to generate an image (including GPU time) on 500 images. We note significant gains in wall-clock time on single-shot image generation with DEQ-DDIM on images with smaller resolutions. Specifically, DEQ-DDIM can generate images almost 2 faster than the sequential sampling of DDIM on CIFAR-10 (3232) and CelebA (6464). We note that these gains vanish on sequences of shorter lengths. This is because the number of fixed point solver iterations needed for convergence becomes comparable to the length of diffusion chain for small values of . Thus, lightweight updates performed on short diffusion chains for sequential sampling are faster compared to compute heavy updates in DEQ-DDIM.
We also report FID scores on DEQ-sDDIM for CIFAR10 in Table 2. We run Anderson solver for a maximum of 50 steps for each image. We observe that while DEQ-sDDIM is slower than DDIM, it always generates images with comparable or better FID scores. For higher levels of stochasticity i.e., for larger valued of , DEQ-sDDIM needs more Anderson solver iterations to converge to a fixed point, which increases image generation wall-clock time. We include additional results in Sec. D.2. Finally, we also find that on full-batch inference with larger batches, sequential sampling might outperform DEQ-DDIM, as the latter would have larger memory requirements in this case, i.e., processing smaller batches of size might be faster than processing larger batches of size .
|FID Scores||Time (in seconds)|
5.3 Model Inversion of DDIM with DEQs
We report the minimum values of squared Frobenius norm between the recovered and target images averaged from 100 different runs in Table 3. We report results for DEQ with (i.e., DEQ-DDIM) in this table, and additional results for (i.e., DEQ-sDDIM) are reported in Figure 17. DEQ outperforms the baseline method on all the datasets by a significant margin. We also plot the training loss curves of DEQ-DDIM and the baseline in Figure 3. We observe that DEQ-DDIM converges faster and has much lower loss values than the baseline method induced by DDIM. We also visualize the images generated with the recovered latent states for DEQ-DDIM in Figure 4 and with DEQ-sDDIM in Figure 5. It is worth noting that images generated with DEQs capture more vivid details of the original images, like textures of foliage, crevices, and other finer details than the baseline. We include additional results of model inversion with DEQ-sDDIM on different datasets in Sec. D.3.
|Min loss||Avg Time (mins)||Min loss||Avg Time (mins)|
6 Related Work
Implicit Deep Learning
Implicit deep learning is an emerging field that introduces structured methods to construct modern neural networks. Different from prior explicit counterparts defined by hierarchy or layer stacking, implicit models take advantage of dynamical systemsKolter et al. (2020); El Ghaoui et al. (2019); Amos (2022), e.g., optimization Amos and Kolter (2017); Wang et al. (2019); Djolonga and Krause (2017); Geng et al. (2021a); Donti et al. (2021), differential equation Chen et al. (2018); Dupont et al. (2019); Song et al. (2021a); Gu et al. (2022), or fixed-point system Bai et al. (2019, 2020); Gu et al. (2020). For instance, Neural ODE Chen et al. (2018) describes a continuous time-dependent system, while Deep Equilibrium (DEQ) model Bai et al. (2019), which is actually path-independent, is a new type of implicit models that outputs the equilibrium states of the underlying system, e.g., from given the input . This fixed-point system can be solved by black-box solvers Anderson (1965); Broyden (1965), and further accelerated by the neural solver Bai et al. (2022b) in the inference. An active topic is the stability Bai et al. (2021); Geng et al. (2021b); Bai et al. (2022a) of such a system as it will gradually deteriorate during training, albeit strong performances Chen et al. (2018); Bai et al. (2019, 2021). DEQ has achieved SOTA results on a wide-range of tasks like language modeling Bai et al. (2019), semantic segmentation Bai et al. (2020), graph modeling Gu et al. (2020); Liu et al. (2021); Park et al. (2021); Chen et al. (2022), object detection Wang et al. (2020), optical flow estimation Bai et al. (2022a), robustness Wei and Kolter (2022); Li et al. (2022b), and generative models like normalizing flow Lu et al. (2021), with theoretical guarantees Winston and Kolter (2020); Kawaguchi (2020); Feng and Kolter (2021); Ling et al. (2022).
Diffusion models Sohl-Dickstein et al. (2015); Ho et al. (2020); Song et al. (2020), or score-based generative models Song and Ermon (2019); Song et al. (2021b), are newly developed generative models that utilize an iterative denoising process to progressively sample from a learned data distribution, which actually is the reverse of a forward diffusion process. They have demonstrated impressive fidelity for text-conditioned image generation Ramesh et al. (2022)
and outperformed state-of-the-art GANs on ImageNetDhariwal and Nichol (2021). Despite the superior practical results, diffusion models suffer from a plodding sampling speed, e.g., over hours to generate 50k CIFAR-sized images Song et al. (2020). To accelerate the sampling from diffusion models, researchers propose to skip a part of the sampling steps by reframing the reverse chain Song et al. (2020); Kong et al. (2021); Kong and Ping (2021), or distill the trained diffusion model into a faster one Luhman and Luhman (2021); Salimans and Ho (2022). Plus, the forward and backward processes in diffusion models can be formulated as stochastic differential equations Song et al. (2021b), bridging diffusion models with Neural ODEs Chen et al. (2018) in implicit deep learning. However, the community still lacks insights into the connection between DEQ and diffusion models, where we build our work to investigate this.
Model inversion gives insights into the latent space of a generative model, as an inability of a generative model to correctly reconstruct an image from its latent code is indicative of its inability to model all the attributes of image correctly. Further, the ability to manipulate the latent codes to edit high-level attributes of images finds applications in many tasks like semantic image manipulation Zhu et al. (2020); Abdal et al. (2021)2021); Li et al. (2022a), in-painting Chung et al. (2021), compressed sensing Bora et al. (2017), etc. For generative models like GANs Goodfellow et al. (2014), inversion is non-trivial and requires alternate methods like learning the mapping from an image to the latent code Bau et al. (2020); Perarnau et al. (2016); Zhu et al. (2016), and optimizing the latent code through optimizers, e.g., both gradient-based Abdal et al. (2019) and gradient-free Huh et al. (2020). For diffusion models like DDPM Ho et al. (2020), the generative process is stochastic, which can make model inversion very challenging. Many existing works based on diffusion models Meng et al. (2021); Chung et al. (2021); Song et al. (2021b); Jalal et al. (2021); Kim and Ye (2021) edit images or solve inverse problems without requiring full model inversion. Instead, they do so by utilizing existing understanding of diffusion models as presented in some recent works Song et al. (2021b); Kadkhodaie and Simoncelli (2020); Kawar et al. (2021). Diffusion models have been widely applied to conditional image generation Choi et al. (2021); Chung et al. (2021); Nichol and Dhariwal (2021); Saharia et al. (2021); Jalal et al. (2021); Sasaki et al. (2021); Kim and Ye (2021); Nichol et al. (2021). Chung et al. (2021) propose a method to reduce the number of steps in reverse conditional diffusion process through better initialization, based on the idea of contraction theory of stochastic differential equations. Our proposed method is orthogonal to this work; we explicitly model DDIM as a joint, multivariate fixed point system and leverage black-box root solvers to solve for the fixed point and also allow for efficient differentiation.
We propose an approach to elegantly unify diffusion models and deep equilibrium (DEQ) models. We model the entire sampling chain of the denoising diffusion implicit model (DDIM) as a joint, multivariate (deep) equilibrium model. This setup replaces the traditional sequential sampling process with a parallel one, thereby enabling us to enjoy speedup obtained from multiple GPUs. Further, we can leverage inexact gradients to optimize the entire sampling chain quickly, which results in significant gains in model inversion. We demonstrate the benefits of this approach on 1) single-shot image generation, where we were able to obtain FID scores on par with or slightly better than those of DDIM; and 2) model inversion, where we achieved much faster convergence. We also propose an easy way to extend DEQ formulation for deterministic DDIM to its stochastic variants. It is possible to further speedup the sampling process by training a DEQ model to predict the noise at a particular timestep of the diffusion chain. We can jointly optimize the noise prediction network, and the latent variables of the diffusion chain, which we leave as future work.
Ashwini Pokle is supported by a grant from the Bosch Center for Artificial Intelligence.
Abdal et al. (2019)
Rameen Abdal, Yipeng Qin, and Peter Wonka.
Image2stylegan: How to embed images into the stylegan latent space?
Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4432–4441, 2019.
- Abdal et al. (2021) Rameen Abdal, Peihao Zhu, Niloy J Mitra, and Peter Wonka. Styleflow: Attribute-conditioned exploration of stylegan-generated images using conditional continuous normalizing flows. ACM Transactions on Graphics (TOG), 40(3):1–21, 2021.
- Amos (2022) Brandon Amos. Tutorial on amortized optimization for learning to optimize over continuous domains. arXiv preprint arXiv:2202.00665, 2022.
Amos and Kolter (2017)
Brandon Amos and J. Zico Kolter.
OptNet: Differentiable optimization as a layer in neural networks.
International Conference on Machine Learning (ICML), 2017.
- Anderson (1965) Donald G Anderson. Iterative procedures for nonlinear integral equations. Journal of the ACM (JACM), 1965.
- Bai et al. (2019) Shaojie Bai, J Zico Kolter, and Vladlen Koltun. Deep equilibrium models. Neural Information Processing Systems (NeurIPS), 2019.
- Bai et al. (2020) Shaojie Bai, Vladlen Koltun, and J Zico Kolter. Multiscale deep equilibrium models. Neural Information Processing Systems (NeurIPS), 2020.
- Bai et al. (2021) Shaojie Bai, Vladlen Koltun, and J Zico Kolter. Stabilizing equilibrium models by jacobian regularization. arXiv preprint arXiv:2106.14342, 2021.
- Bai et al. (2022a) Shaojie Bai, Zhengyang Geng, Yash Savani, and J Zico Kolter. Deep equilibrium optical flow estimation. arXiv preprint arXiv:2204.08442, 2022a.
- Bai et al. (2022b) Shaojie Bai, Vladlen Koltun, and J Zico Kolter. Neural deep equilibrium solvers. In International Conference on Learning Representations, 2022b.
- Bau et al. (2020) David Bau, Hendrik Strobelt, William Peebles, Jonas Wulff, Bolei Zhou, Jun-Yan Zhu, and Antonio Torralba. Semantic photo manipulation with a generative image prior. arXiv preprint arXiv:2005.07727, 2020.
- Bora et al. (2017) Ashish Bora, Ajil Jalal, Eric Price, and Alexandros G Dimakis. Compressed sensing using generative models. In International Conference on Machine Learning, pages 537–546. PMLR, 2017.
- Broyden (1965) Charles G Broyden. A class of methods for solving nonlinear simultaneous equations. Mathematics of computation, 1965.
Chan et al. (2021)
Kelvin CK Chan, Xintao Wang, Xiangyu Xu, Jinwei Gu, and Chen Change Loy.
Glean: Generative latent bank for large-factor image
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14245–14254, 2021.
- Chen et al. (2022) Qi Chen, Yifei Wang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Optimization-induced graph implicit nonlinear diffusion. In International Conference on Machine Learning, pages 3648–3661. PMLR, 2022.
Chen et al. (2018)
Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud.
Neural ordinary differential equations.In Neural Information Processing Systems (NeurIPS), 2018.
- Choi et al. (2021) Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. Ilvr: Conditioning method for denoising diffusion probabilistic models. arXiv preprint arXiv:2108.02938, 2021.
- Chung et al. (2021) Hyungjin Chung, Byeongsu Sim, and Jong Chul Ye. Come-closer-diffuse-faster: Accelerating conditional diffusion models for inverse problems through stochastic contraction. arXiv preprint arXiv:2112.05146, 2021.
- Dhariwal and Nichol (2021) Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34, 2021.
- Djolonga and Krause (2017) Josip Djolonga and Andreas Krause. Differentiable learning of submodular models. Advances in Neural Information Processing Systems, 30, 2017.
- Donti et al. (2021) Priya L. Donti, David Rolnick, and J Zico Kolter. DC3: A learning method for optimization with hard constraints. In International Conference on Learning Representations (ICLR), 2021.
- Dupont et al. (2019) Emilien Dupont, Arnaud Doucet, and Yee Whye Teh. Augmented neural ODEs. In Neural Information Processing Systems (NeurIPS), 2019.
- El Ghaoui et al. (2019) Laurent El Ghaoui, Fangda Gu, Bertrand Travacca, and Armin Askari. Implicit deep learning. arXiv:1908.06315, 2019.
- Falk et al. (2019) Thorsten Falk, Dominic Mai, Robert Bensch, Özgün Çiçek, Ahmed Abdulkadir, Yassine Marrakchi, Anton Böhm, Jan Deubner, Zoe Jäckel, Katharina Seiwald, et al. U-net: deep learning for cell counting, detection, and morphometry. Nature methods, 16(1):67–70, 2019.
- Feng and Kolter (2021) Zhili Feng and J Zico Kolter. On the neural tangent kernel of equilibrium models, 2021.
- Fung et al. (2021) Samy Wu Fung, Howard Heaton, Qiuwei Li, Daniel McKenzie, Stanley Osher, and Wotao Yin. Fixed point networks: Implicit depth models with jacobian-free backprop. arXiv e-prints, pages arXiv–2103, 2021.
- Geng et al. (2021a) Zhengyang Geng, Meng-Hao Guo, Hongxu Chen, Xia Li, Ke Wei, and Zhouchen Lin. Is attention better than matrix decomposition? In International Conference on Learning Representations (ICLR), 2021a.
- Geng et al. (2021b) Zhengyang Geng, Xin-Yu Zhang, Shaojie Bai, Yisen Wang, and Zhouchen Lin. On training implicit models. Neural Information Processing Systems (NeurIPS), 2021b.
- Goodfellow et al. (2014) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
- Gu et al. (2022) Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations (ICLR), 2022.
- Gu et al. (2020) Fangda Gu, Heng Chang, Wenwu Zhu, Somayeh Sojoudi, and Laurent El Ghaoui. Implicit Graph Neural Networks. In Neural Information Processing Systems (NeurIPS), pages 11984–11995, 2020.
- Heusel et al. (2017) Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
- Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Neural Information Processing Systems (NeurIPS), 2020.
- Huh et al. (2020) Minyoung Huh, Richard Zhang, Jun-Yan Zhu, Sylvain Paris, and Aaron Hertzmann. Transforming and projecting images into class-conditional generative networks. In European Conference on Computer Vision, pages 17–34. Springer, 2020.
- Issenhuth et al. (2021) Thibaut Issenhuth, Ugo Tanielian, Jérémie Mary, and David Picard. Edibert, a generative model for image editing. arXiv preprint arXiv:2111.15264, 2021.
- Jalal et al. (2021) Ajil Jalal, Marius Arvinte, Giannis Daras, Eric Price, Alexandros G Dimakis, and Jon Tamir. Robust compressed sensing mri with deep generative priors. Advances in Neural Information Processing Systems, 34:14938–14954, 2021.
- Kadkhodaie and Simoncelli (2020) Zahra Kadkhodaie and Eero P Simoncelli. Solving linear inverse problems using the prior implicit in a denoiser. arXiv preprint arXiv:2007.13640, 2020.
- Kawaguchi (2020) Kenji Kawaguchi. On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layers. In International Conference on Learning Representations (ICLR), 2020.
- Kawar et al. (2021) Bahjat Kawar, Gregory Vaksman, and Michael Elad. Snips: Solving noisy inverse problems stochastically. Advances in Neural Information Processing Systems, 34:21757–21769, 2021.
- Kim and Ye (2021) Gwanghyun Kim and Jong Chul Ye. Diffusionclip: Text-guided image manipulation using diffusion models. arXiv preprint arXiv:2110.02711, 2021.
- Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
- Kingma et al. (2021) Diederik P Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. arXiv preprint arXiv:2107.00630, 2021.
- Kolter et al. (2020) J. Zico Kolter, David Duvenaud, and Matthew Johnson. Deep implicit layers tutorial - neural ODEs, deep equilibirum models, and beyond. Neural Information Processing Systems Tutorial, 2020.
- Kong and Ping (2021) Zhifeng Kong and Wei Ping. On fast sampling of diffusion probabilistic models. arXiv preprint arXiv:2106.00132, 2021.
- Kong et al. (2021) Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations (ICLR), 2021.
Learning multiple layers of features from tiny images.
Technical report, Citeseer, 2009.
- Li et al. (2022a) Haoying Li, Yifan Yang, Meng Chang, Shiqi Chen, Huajun Feng, Zhihai Xu, Qi Li, and Yueting Chen. Srdiff: Single image super-resolution with diffusion probabilistic models. Neurocomputing, 2022a.
- Li et al. (2022b) Mingjie Li, Yisen Wang, and Zhouchen Lin. Cerdeq: Certifiable deep equilibrium model. In International Conference on Machine Learning, 2022b.
- Ling et al. (2021) Huan Ling, Karsten Kreis, Daiqing Li, Seung Wook Kim, Antonio Torralba, and Sanja Fidler. Editgan: High-precision semantic image editing. Advances in Neural Information Processing Systems, 34:16331–16345, 2021.
- Ling et al. (2022) Zenan Ling, Xingyu Xie, Qiuhao Wang, Zongpeng Zhang, and Zhouchen Lin. Global convergence of over-parameterized deep equilibrium models. arXiv preprint arXiv:2205.13814, 2022.
- Liu et al. (2021) Juncheng Liu, Kenji Kawaguchi, Bryan Hooi, Yiwei Wang, and Xiaokui Xiao. EIGNN: Efficient infinite-depth graph neural networks. In Neural Information Processing Systems (NeurIPS), 2021.
- Liu et al. (2015) Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
- Lu et al. (2021) Cheng Lu, Jianfei Chen, Chongxuan Li, Qiuhao Wang, and Jun Zhu. Implicit normalizing flows. In International Conference on Learning Representations (ICLR), 2021.
- Luhman and Luhman (2021) Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed. ArXiv, abs/2101.02388, 2021.
- Meng et al. (2021) Chenlin Meng, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.01073, 2021.
- Nichol et al. (2021) Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741, 2021.
- Nichol and Dhariwal (2021) Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pages 8162–8171. PMLR, 2021.
- Nie et al. (2022) Weili Nie, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, and Anima Anandkumar. Diffusion models for adversarial purification. arXiv preprint arXiv:2205.07460, 2022.
- Park et al. (2021) Junyoung Park, Jinhyun Choo, and Jinkyoo Park. Convergent graph solvers. arXiv preprint arXiv:2106.01680, 2021.
- Paszke et al. (2017) Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
- Perarnau et al. (2016) Guim Perarnau, Joost Van De Weijer, Bogdan Raducanu, and Jose M Álvarez. Invertible conditional gans for image editing. arXiv preprint arXiv:1611.06355, 2016.
- Ramesh et al. (2022) Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
- Saeedi et al. (2018) Ardavan Saeedi, Matthew Hoffman, Stephen DiVerdi, Asma Ghandeharioun, Matthew Johnson, and Ryan Adams. Multimodal prediction and personalization of photo edits with deep generative models. In International Conference on Artificial Intelligence and Statistics, pages 1309–1317. PMLR, 2018.
- Saharia et al. (2021) Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. arXiv preprint arXiv:2104.07636, 2021.
- Salimans and Ho (2022) Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations (ICLR), 2022.
- Sasaki et al. (2021) Hiroshi Sasaki, Chris G Willcocks, and Toby P Breckon. Unit-ddpm: Unpaired image translation with denoising diffusion probabilistic models. arXiv preprint arXiv:2104.05358, 2021.
Sohl-Dickstein et al. (2015)
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.
Deep unsupervised learning using nonequilibrium thermodynamics.In International Conference on Machine Learning (ICML), 2015.
- Song et al. (2020) Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
- Song and Ermon (2019) Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
- Song et al. (2021a) Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations (ICLR), 2021a.
- Song et al. (2021b) Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations (ICLR), 2021b.
- Wang et al. (2019) Po-Wei Wang, Priya Donti, Bryan Wilder, and Zico Kolter. Satnet: Bridging deep learning and logical reasoning using a differentiable satisfiability solver. In International Conference on Machine Learning (ICML), 2019.
- Wang et al. (2020) Tiancai Wang, Xiangyu Zhang, and Jian Sun. Implicit Feature Pyramid Network for Object Detection. arXiv preprint arXiv:2012.13563, 2020.
- Wei and Kolter (2022) Colin Wei and J Zico Kolter. Certified robustness for deep equilibrium models via interval bound propagation. In International Conference on Learning Representations, 2022.
- Winston and Kolter (2020) Ezra Winston and J. Zico Kolter. Monotone operator equilibrium networks. In Neural Information Processing Systems (NeurIPS), 2020.
- Yu et al. (2015) Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. LSUN: construction of a large-scale image dataset using deep learning with humans in the loop. CoRR, abs/1506.03365, 2015.
- Zhu et al. (2020) Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou. In-domain gan inversion for real image editing. In European conference on computer vision, pages 592–608. Springer, 2020.
- Zhu et al. (2016) Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A Efros. Generative visual manipulation on the natural image manifold. In European conference on computer vision, pages 597–613. Springer, 2016.
Appendix A Experimental Details
In this section, we present detailed settings for all the experiments in Section 5.
We follow the linear selection procedure to select a subsequence of timesteps for all the datasets except CIFAR10, i.e., we select timesteps such that for some . For CIFAR10, we select timesteps such that for some . The constant is selected so that is close to . We use Anderson acceleration Anderson , Kolter et al.  as our fixed-point solver for all the experiments. We set the exiting equilibrium error of solver to 1e-3 and set the history length to 5. We allow a maximum of 15 solver forward steps in all the experiments with DEQ-DDIM, and use a maximum of 50 solver forward steps for DEQ-sDDIM. Finally, we use PyTorch’s inbuilt DataParallel module to handle parallelization. We use upto 4 NVIDIA Quadro RTX 8000 or RTX A6000 GPUs for all our experiments.
Training details for model inversion
We implement and test the code in PyTorch version 1.11.0. We use the Adam Kingma and Ba  optimizer with a learning rate of 0.01. We train DEQs for 400 epochs on CIFAR10 and CelebA, and for 500 epochs on LSUN Bedroom, and LSUN Outdoor Church. The baseline is trained for 1000 epochs on CIFAR10 with , for 3000 epochs on CIFAR10 with , for 2500 epochs on CelebA, and for 2000 epochs on LSUN Bedroom, and LSUN Outdoor Church. At the beginning of inversion procedure with DEQ-DDIM, we sample , and initialize the latents at all (or the subsequence of) timesteps to this value. For inversion with DEQ-sDDIM we also sample . We stop the training as soon as the loss falls below 0.5 for CIFAR10, and below 2 for other datasets.
We compute FID scores using the code provided by https://github.com/w86763777/pytorch-gan-metrics. We also use the precomputed statistics for CIFAR10, LSUN Bedrooms and Outdoor Churches provided in this github repository. For CelebA, we compute our own dataset statistics, as the precomputed statistics for images of resolution 6464 are not included in this repository. While computing the statistics, we preprocess the images of CelebA in exactly the same way as done by Song et al. .
Appendix B Pseudocode
Appendix C Ablation Studies for Model Inversion with DEQ-DDIM
c.1 Effect of length of sampling chain
Effect of length of subsequence during training
We study the effect of the length of the diffusion chain on the convergence rate of optimization for model inversion in Fig. 6. We note that for sequential sampling, loss decreases slightly faster for the smaller diffusion chain ( and ) than the longer one () for the baseline. However, for DEQ-DDIM, the length of diffusion chain doesn’t seem to have an effect on the rate of convergence as the loss curves for and and are nearly identical.
Effect of length of subsequence during sampling
All the images in Fig. 4 are sampled with a subsequence of timesteps , i.e., the number of latents in the diffusion chain used for training and the number of timesteps used for sampling an image from the recovered were equal. We investigate if sampling with results in images with a better perceptual quality for the baseline. We display the recovered images for LSUN Bedrooms in Fig. 7. The length of diffusion chain during training time is . We note that using more sampling steps does not result in inverted images that are closer to the original image. In some cases, samples generated with more sampling steps have some additional artifacts that are not present in the original image.
Comparing exact vs inexact gradients for backward pass of DEQ-DDIM
The choice of gradient calculation for the backward pass of DEQ affects both the training stability and convergence of DEQ-DDIM. Here, we compare the performance of the exact gradients and inexact gradients. Computing the inverse of Jacobian in Eq. 15 is difficult because the Jacobian can be prohibitively large. We follow Bai et al.  to compute exact gradients using the following linear system
We use Broyden’s method [Broyden, 1965] to efficiently solve for in this linear system. We compare it againt inexact gradients i.e., Jacobian free gradient used in Algorithm 2. We observe that training DEQ-DDIM with exact gradients becomes increasingly unstable as the training proceeds, especially for larger learning rates like 0.005. However, we can converge faster with larger learning rates like 0.01 with inexact gradients.
Effects of choice of initialization on convergence of DEQ-DDIM
The choice of initialization is critical for fast convergence of DEQs. Bai et al.  initialize the initial estimate of fixed point of DEQ with zeros. However, in this work, we initialize all the latent states with , as it results in faster convergence shown in Figure 9. While both the initialization schemes converge to high quality images eventually, we observe that initializing with results in up to faster convergence compared to zero initialization. We observe a significant qualitative difference in the visualization of the intermediate states of the diffusion chain at different solver steps for the two initialization schemes as observed in Figure 10.
Appendix D Extending DEQ formulation to stochastic DDIM (DEQ-sDDIM)
A more general and highly stochastic generative process is to sample and integrate noises every time step, given by [Song et al., 2020]:
where . Here, corresponds to a deterministic DDIM while corresponds to a stochastic DDIM. Empirically, this is parameterized as where
is a hyperparameter to control stochasticity. Note thatcorresponds to a DDPM Sohl-Dickstein et al. , Ho et al. .
Rearranging the terms in Eq. (20), we get
Let . Then we can write
By induction, we can rewrite the above equation as:
This again defines a “fully-lower-triangular” inference process, where the update of depends on the noise prediction network applied to all subsequent states .
Following the notation used in the main paper, let represent the function that performs the operations in the equations (23) for a latent at timestep , let represent the function that performs the same set of operations across all the timesteps simultaneously, and let represent the noise injected into the diffusion process at every timestep. We can write the above set of equations as a fixed point system:
The above system of equations represent a DEQ with and as input injection. We refer to this formulation of DEQ for stochastic DDIM with as DEQ-sDDIM.
A major difference between the both is that DEQ-sDDIM can exploit the noises sampled prior to fixed point solving as addition input injections. The insight here is that the noises along the sampling chain are independent of each other, thus allowing us to sample all the noises simultaneously and convert a highly stochastic autoregressive sampling process into a deterministic “fully-lower-triangular” DEQ.
Then we can now use black-box solvers like Anderson acceleration [Anderson, 1965] to solve for the roots of this system of equations as the previous case. Let , then we have
where RootSolver() is any black-box fixed point solver.
d.1 Convergence of DEQ-sDDIMs
We verify that our DEQ version for stochastic DDIM converges to a fixed point by plotting values of over Anderson solver steps in Figure 11. As one would expect, we need more solver steps to solve for the fixed point given higher values of .
d.2 Sample quality of images generated with DEQ-sDDIM
We verify that DEQ-sDDIM can generate images that are on par with the original DDIM by computing FID scores on 50,000 sampled images. We report our results in Table 2 and Table 4. We observe that our FID scores are comparable or slightly better to those from sequential DDIM. We also visualize images generated from the same latent at different levels of stochasticity controlled through . We display our generated images in Figure 12 and Figure 13.
|FID Scores||Time (in seconds)|
d.3 Model inversion with DEQ-sDDIM
We report the minimum values of squared Frobenius norm between the recovered and target images averaged from 25 different runs in Figure 17. We use the same hyperparameters as the ones used for training DEQ models for DDIM in these experiments. DEQ-sDDIM is able to achieve low values of the reconstruction loss even for large values of like as noted in Figure 17. We also plot training loss curves for different values of in Figure 17 on CIFAR10. We note that it indeed takes longer time to invert DEQ-sDDIM for higher values of . This is primarily because the fixed point solver needs more iterations to converge. However, despite that we obtain impressive model inversion results on CIFAR10 and CelebA. We visualize images generated with the recovered latent states in Figure 14 and Figure 15.