The Spectral Bias of the Deep Image Prior

by   Prithvijit Chakrabarty, et al.

The "deep image prior" proposed by Ulyanov et al. is an intriguing property of neural nets: a convolutional encoder-decoder network can be used as a prior for natural images. The network architecture implicitly introduces a bias; If we train the model to map white noise to a corrupted image, this bias guides the model to fit the true image before fitting the corrupted regions. This paper explores why the deep image prior helps in denoising natural images. We present a novel method to analyze trajectories generated by the deep image prior optimization and demonstrate: (i) convolution layers of the an encoder-decoder decouple the frequency components of the image, learning each at different rates (ii) the model fits lower frequencies first, making early stopping behave as a low pass filter. The experiments study an extension of Cheng et al which showed that at initialization, the deep image prior is equivalent to a stationary Gaussian process.


page 2

page 6

page 7


Image denoising and restoration with CNN-LSTM Encoder Decoder with Direct Attention

Image denoising is always a challenging task in the field of computer vi...

A Bayesian Perspective on the Deep Image Prior

The deep image prior was recently introduced as a prior for natural imag...

Approximated Infomax Early Stopping: Revisiting Gaussian RBMs on Natural Images

We pursue an early stopping technique that helps Gaussian Restricted Bol...

On Measuring and Controlling the Spectral Bias of the Deep Image Prior

The deep image prior has demonstrated the remarkable ability that untrai...

Denoising and Regularization via Exploiting the Structural Bias of Convolutional Generators

Convolutional Neural Networks (CNNs) have emerged as highly successful t...

Rethinking Deep Image Prior for Denoising

Deep image prior (DIP) serves as a good inductive bias for diverse inver...

Contextualize, Show and Tell: A Neural Visual Storyteller

We present a neural model for generating short stories from image sequen...

Code Repositories

1 Introduction

It is well known that large neural nets have the capacity to overfit training data, even fitting random labels perfectly Zhang et al. (2016). Arpit et al. Arpit et al. (2017) confirmed this, but showed that networks learn "simple patterns" first. This may explain why these models, despite their capacity, generalize well with early stopping. Neyshabur et al. (2017); Gunasekar et al. (2018); Soudry et al. (2018); Neyshabur et al. (2014). One way to formalize the notion of a simple pattern is to observe the frequency components of the function learned by the model. Simple patterns are smooth, composed of low frequencies. A number of works use this approach to describe the bias induced by deep networks. For example, Xu et al. (2018, 2019) study the F-principle which states that the models first learn the low-frequency components when fitting a signal. Rahaman et al. Rahaman et al. (2018) demonstrated a similar spectral bias

of a deep fully-connected network with ReLU activation.

The Deep Image Prior (DIP) considers the following setup. Let be a convolutional encoder-decoder parameterized by . , are spaces of dimensional signals. DIP studies the following optimization: where, is a fixed

dimensional white noise vector.

steps of gradient descent for this optimization traces out a trajectory in the parameter space: . This has a corresponding trajectory in the output space : . Given enough capacity and suitable learning rate scheme, for a large , the model will perfectly fit the signal, i.e., . For image denoising , are spaces of all images () and is a noisy image: a clean image with added Gaussian noise . Experiments in Ulyanov et al. (2018) show that early stopping with gradient descent will lead to denoised the image. In other words, the trajectory will contain a point that is close to the clean image .

We use the spectral bias of the network to explain this denoising behavior. It is known that at initialization the generated output of the DIP is drawn from distribution that is approximately a stationary Gaussian process with smooth covariance function Cheng et al. (2019). The experiments presented here suggest that this trend continues throughout the optimization, i.e., the model learns to construct the image from low to high frequencies. Thus, early stopping prevents fitting the high frequency components introduced by the additive Gaussian noise. The source code to reproduce results is available online111

(a) (b) (c) (d) (e) (f) (g) (h)
Figure 1: (a-e) DIP applied to two clean images. (a) Input images. (b) Initial output. (c) Output at the first convergence of trajectories. (d) Output at the second convergence of trajectories. (e) Variation of across iterations. (f-g) Controlled amounts of high frequency patterns on triangle.png. Trajectories diverge more when the pattern covers a larger area. (Please zoom in for details.)

2 The Spectral Bias of the Deep Image Prior

We first demonstrate the spectral bias of the DIP in the exact setting as Ulyanov et al. Ulyanov et al. (2018)

. Using the same model, i.e., a 5-layered convolutional autoencoder with

channels each, the DIP optimization was run on clean images which have a range of frequency components, as shown in Figure 1(a). For each image, the optimization was run twice to generate trajectories , in the output space. Let be the variation of the sum of squared error (SSE) between trajectories: .

(a) (b) (c)
Figure 2: Some failure cases for the DIP. (a) Images with low frequency noise. (b) iterations: the model fits the noise, not the true high frequency components. (c) iterations: the model fits the signal perfectly.

Fig. 1 tracks the on two images: barbara.png naturally has high frequency components while triangle.png is a smooth image with a high frequency pattern superimposed on it. We observe the following:

  • [label=•,nosep,leftmargin=*]

  • The initial output of the model is smooth (Fig. 1b). This is consistent with the analysis in Cheng et al. (2019), which showed that at initialization the output can be approximated as a draw from a stationary Gaussian process with a smooth covariance function.

  • The trajectories converge twice (Fig. 1e). At the first point of convergence, the model predicts a low frequency reconstruction of the image (Fig. 1c). After the second converge, the model fits the high frequency components (Fig. 1d).

  • Controlled amounts of the high frequency pattern were added to triangle.png. As it covers a larger spatial extent, the trajectories diverge more (Fig. 1(f-h)). This suggests that the reason for divergence corresponds to the model learning the higher frequencies.

These observations show that the model learns the frequency components of the image at different rates, fitting the low frequencies first (see Appendix A

). Thus, early stopping is similar to low-pass filtering in the frequency domain.

The denoising experiments in Ulyanov et al. (2018) used additive Gaussian noise. Early stopping with DIP prevents fitting these, predicting the clean image before fitting the noise. We can now construct samples where deep image prior is guaranteed to fail. Low frequency noise is added to barbara.png and baboon.png: images which naturally have high frequencies, as shown in Fig. 2. After iterations, the model fits the noise, but not the high frequency components and goes on the fit the input perfectly after iterations. Thus, there is no point at which the model predicts the clean image. This strongly suggests that the ability of DIP to denoise images is brought about due to the frequency bias.

3 What causes the frequency bias?

Here we investigate what elements of the DIP lead to the aforementioned frequency bias in learning. In particular we show that both convolutions and upsampling introduce a bias.

(a) (b) (c) (d)
(e) (f) (g) (h)
Figure 3: DIP on 1D signals. (a-e) Outputs of a model with conv layers. (a) Input signal (b) Predicted signal when converges, (c) when converges and (d) at the end of the optimization. (e) Variation of squared errors and across iterations. (f-h) Outputs of a model with linear layers. Variation of and with (f) 256 nodes per layers (g) 4096 nodes per layers (h) depth 24. This model fails to decouple frequencies.

3.1 Convolution Layers

The frequency selectiveness of a convolutional network architecture was shown in Saxe et al. (2011). DIP uses a convolutional encoder-decoder. Here, we demonstrate using a simple experiment that the convolution layers of this architecture decouple the frequencies of the signal, fitting each at different rates. This does not happen if the encoder-decoder has only linear layers. Consider DIP on 1D signals, where . The signal with and is shown in fig. 3(a). We run the DIP optimization on this signal and track the squared error , where is the predicted amplitude and is the true amplitude for frequency . We say that a frequency has converged at if after iterations ( for these experiments). This is similar to Experiment 1 in Rahaman et al. (2018). We compare the use of convolutions against linear layers in the model:

  • [label=•,nosep,leftmargin=*]

  • DIP-Conv. A 10-layered encoder-decoder with 1D convolution layers, 256 channels. The variation in and is shown in Fig. 3(e). drops sharply, leading to converge at 45 iterations. is learned slowly, converging at 151 iterations. Observing the predictions after each component converges (Fig. 3(b,c)), we see when converges, the model predicts a smooth reconstruction of the signal. Clearly, the frequency components are learned at different rates.

  • DIP-Linear. The convolution layers of the DIP-conv were replaced with 256 unit linear layers (fully-connected) and the optimization was run again. Fig. 3(f) shows the results. The error for both frequencies drop at the same rate. The model never predicts a low-frequency reconstruction of the signal. Further, the lack of decoupling remains if we change the depth or width of the network (Fig. 3(g,h)) which suggests that it is intrinsic to linear layers and not related to the model capacity.

27.47 20.54 19.17 27.58
Table 1: Denoising performance. Only models with frequency bias are effective at denoising.

Effect on image denoising. To confirm that the above results extend to 2D signals, the same architecture variations were applied to images. DIP: Standard DIP for images using the model from Ulyanov et al. (2018) (similar to DIP-Conv above); DIP Linear-128: An encoder-decoder with 5 fully-connected layers, units each; DIP Linear-2048: DIP Linear, with units each per layer (higher capacity); ReLUNet: A 10-layered fully-connected network with nodes per layer and ReLU activations to model images as signals. The model is trained to map pixel coordinates to the corresponding intensities (this will exhibit frequency bias as shown in Rahaman et al. (2018)).

The ReLUNet and DIP-Linear architectures do not explicitly have convolution layers. However, their behavior is closely related to special cases of the DIP model. A ReLUNet predicts individual pixels without using neighborhood information, similar to a convolutional encoder-decoder with a kernel size of 1. DIP-Linear is similar to the other extreme, when the kernel size equals the size of the entire signal.

These models were used to fit images used in Ulyanov et al. (2018); Cheng et al. (2019) from a standard dataset222 For efficiency, the images were downsampled by a factor of . The entire trajectories in the output space were saved to see if they contain the denoised image (samples in Appendix D). To eliminate the effect of a badly chosen stopping time, we track the PSNR across the entire trajectory and record the best.

Table 1 shows the mean PSNR across 9 images (See appendix C for results per image). DIP performs the best, with ReLUNets achieving comparable performance. Encoder-decoders with only fully-connected layers perform significantly worse, irrespective of capacity, as they do not decouple frequencies. These results reinforce the idea that the denoising ability is a result of the frequency bias.

3.2 Upsampling Layers

(a) (b)
(c) (d)
Figure 4: (a-b) Error in the power spectrum across the first

iterations using (a) stride=

and (b) stride=. (c-d) Output at iterations with (c) stride= and (d) stride=. A larger stride produces a smoother output at the same iteration.

Two types of upsampling methods were explored in Ulyanov et al. (2018): nearest neighbor and bilinear. Both methods can be viewed as upsampling with zeros followed by a convolution with a fixed smoothing kernel. These strongly bias the model towards smooth images. For example, upsampling a 1D signal with stride , the frequency responses of the associated convolution operations are for nearest-neighbor. and for bilinear upsampling (Appendix E). The responses decay with frequency and stride. This is qualitatively demonstrated in Fig. 4. The low error region of the power spectrum (dark region at the bottom of Fig. 4(a,b)) grows more slowly, with a higher stride.

4 Conclusion

This paper studies the DIP, proposing an explanation for the denoising experiments presented in Ulyanov et al. (2018). We associate the phenomenon with the notion of frequency bias: convolutional networks fit the low frequencies of a signal first. Experiments on 1D signals and images with additive Gaussian noise strongly suggest that this is the causative factor behind the denoising behavior observed in DIP.


  • D. Arpit, S. Jastrzebski, N. Ballas, D. Krueger, E. Bengio, M. S. Kanwal, T. Maharaj, A. Fischer, A. Courville, Y. Bengio, et al. (2017) A closer look at memorization in deep networks. In

    Proceedings of the 34th International Conference on Machine Learning-Volume 70

    pp. 233–242. Cited by: §1.
  • Z. Cheng, M. Gadelha, S. Maji, and D. Sheldon (2019) A bayesian perspective on the deep image prior. arXiv preprint arXiv:1904.07457. Cited by: The Spectral Bias of the Deep Image Prior, §1, 1st item, §3.1.
  • S. Gunasekar, J. D. Lee, D. Soudry, and N. Srebro (2018) Implicit bias of gradient descent on linear convolutional networks. In Advances in Neural Information Processing Systems, pp. 9461–9471. Cited by: §1.
  • B. Neyshabur, S. Bhojanapalli, D. McAllester, and N. Srebro (2017)

    Exploring generalization in deep learning

    In Advances in Neural Information Processing Systems, pp. 5947–5956. Cited by: §1.
  • B. Neyshabur, R. Tomioka, and N. Srebro (2014) In search of the real inductive bias: on the role of implicit regularization in deep learning. arXiv preprint arXiv:1412.6614. Cited by: §1.
  • N. Rahaman, D. Arpit, A. Baratin, F. Draxler, M. Lin, F. A. Hamprecht, Y. Bengio, and A. Courville (2018)

    On the spectral bias of deep neural networks

    arXiv preprint arXiv:1806.08734. Cited by: §1, §3.1, §3.1.
  • A. M. Saxe, P. W. Koh, Z. Chen, M. Bhand, B. Suresh, and A. Y. Ng (2011) On random weights and unsupervised feature learning.. In ICML, Vol. 2, pp. 6. Cited by: §3.1.
  • D. Soudry, E. Hoffer, M. S. Nacson, S. Gunasekar, and N. Srebro (2018) The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research 19 (1), pp. 2822–2878. Cited by: §1.
  • D. Ulyanov, A. Vedaldi, and V. Lempitsky (2018) Deep image prior. In

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

    pp. 9446–9454. Cited by: The Spectral Bias of the Deep Image Prior, §1, §2, §2, §3.1, §3.1, §3.2, §4.
  • Z. J. Xu, Y. Zhang, and Y. Xiao (2018) Training behavior of deep neural network in frequency domain. arXiv preprint arXiv:1807.01251. Cited by: §1.
  • Z. J. Xu, Y. Zhang, T. Luo, Y. Xiao, and Z. Ma (2019) Frequency principle: fourier analysis sheds light on deep neural networks. arXiv preprint arXiv:1901.06523. Cited by: §1.
  • C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals (2016) Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530. Cited by: §1.

5 Appendix

A Power Spectrum in Experiment 1

Fig. 5 shows the trajectory in Exp. 1 Fig. 1) and the corresponding power spectrum. This clearly shows the frequency components growing from low to high frequencies.

Figure 5: The power spectrum of DIP output on clean images. Iterations , , , and for (a) barbara.png and (b) triangle.png. (Please zoom in to see the patterns clearly.)

B Effect of model capacity

We study the effect of changing the model capacity in the experiments with 1D signals with 2 frequency components (Section 3.1). The depth and width of the model was varied, keeping all other settings fixed. For each capacity, we record the mean and standard deviation of the time to convergence for each frequency component across


Fig. 6(a,b) show the results when the model uses only convolution layers. We observe the following:

  • [label=•,nosep,leftmargin=*]

  • Increasing the depth makes the model separate frequencies more. In Fig. 6(a), as the depth increases, the mean convergence time for increases much faster than that for , indicating that the model fits the higher frequency more slowly.

  • Increasing the width (number of channels per layer) leads to faster convergence for both frequencies (fig. 6(b)). The convergence time of drops faster, eventually becoming lower than that of ( channels per layer generally fits the higher frequency first).

These results suggest that a deep, narrow model will be more effective at decoupling frequencies than a wide, shallow one. Fig. 6(c,d) show the results when the model uses only fully-connected layers. The overall trend is the same as that of the convolutional model: time to convergence increases with depth and decreases with width. However, both frequency components converge at the same time. This supports the conclusion that fully-connected layers are unable to decouple frequencies.

(a) (b) (c) (d)
Figure 6: Effect of varying capacity on the convergence time of frequency components and . (a-b) Results with convolution layers: convergence time with (a) increasing depth (b) increasing number of channels per layer. (c-d) Results with linear layers: convergence time with (c) increasing depth (d) increasing number of nodes per hidden layer.

C Detailed Denoising Results

Denoising results on each image using the architectures in section 3.1. Table 1 summarizes the table below, showing the means per column.

Image DIP DIP Linear-128 DIP Linear-2048 ReLUNet
Baboon 24.730.04 20.480.06 20.350.43 24.560.02
F16 27.000.15 20.720.12 18.601.85 26.670.14
House 26.890.10 20.690.04 18.891.45 27.210.22
kodim01 25.900.08 20.350.07 20.540.07 25.950.07
kodim02 29.200.13 20.740.01 20.590.19 29.990.09
kodim03 29.260.13 20.470.04 18.130.13 29.620.16
kodim12 29.610.06 20.360.03 17.740.04 29.630.16
Lena 27.490.06 20.480.01 18.670.72 27.780.16
Peppers 27.120.11 20.600.01 18.981.31 26.820.16
Table 2: Per-image denoising performance. Mean and standard deviation of the PSNR across runs.

D Trajectories of denoising experiment

Fig. 7 shows samples from the trajectories while denoising with DIP (fig. 7(a)) and with a ReLUNet (fig. 7(b)). The input image is House.png, downsampled by a factor of (to x) with added Gaussian noise. Both models demostrate frequency bias and predict the clean image before fitting the noise. The samples also show the difference between the DIP and ReLUNet models. ReLUNets are extremely biased towards smooth images, taking much longer to fit the higher frequencies: the DIP model starts fitting the noise at iterations, while the ReLUNet model does so at iterations (even then, the model doesn’t fit these components perfectly, predicting a blurred version of the noise).

(a) (b)
Figure 7: Samples from trajectories when denoising House.png small. (a) Output of DIP model at iterations , , (b) Output of ReLUNet model at iterations , ,

E Frequency response of upsampling methods

Standard upsampling methods can be viewed as upsampling by padding with zeros followed by a convolution with a fixed smoothing kernel

. Let be the frequency response of this kernel (where is the frequency). Upsampling with zeros scales the image by a factor , but adds high frequency components. These components are removed after convolution with if decays with frequency. The spatial extent of is the upsampling stride . For example, in 1D, these kernels are as follows:

  • [label=•,nosep,leftmargin=*]

  • For nearest neighbor, the kernel is the box function:
    The frequency response of this kernel is:

  • For bilinear upsampling, the kernel is the triangle wave function:
    The frequency response of this kernel is:

Clearly, both favor low frequency components, with the bilinear having a stronger bias () than nearest neighbor ().