Scalable Adaptive Computation for Iterative Generation

12/22/2022
by   Allan Jabri, et al.
0

We present the Recurrent Interface Network (RIN), a neural net architecture that allocates computation adaptively to the input according to the distribution of information, allowing it to scale to iterative generation of high-dimensional data. Hidden units of RINs are partitioned into the interface, which is locally connected to inputs, and latents, which are decoupled from inputs and can exchange information globally. The RIN block selectively reads from the interface into latents for high-capacity processing, with incremental updates written back to the interface. Stacking multiple blocks enables effective routing across local and global levels. While routing adds overhead, the cost can be amortized in recurrent computation settings where inputs change gradually while more global context persists, such as iterative generation using diffusion models. To this end, we propose a latent self-conditioning technique that "warm-starts" the latents at each iteration of the generation process. When applied to diffusion models operating directly on pixels, RINs yield state-of-the-art image and video generation without cascades or guidance, while being domain-agnostic and up to 10× more efficient compared to specialized 2D and 3D U-Nets.

READ FULL TEXT

page 6

page 7

page 13

page 14

page 15

page 16

page 17

research
11/02/2022

eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers

Large-scale diffusion-based generative models have led to breakthroughs ...
research
04/13/2023

Learning Controllable 3D Diffusion Models from Single-view Images

Diffusion models have recently become the de-facto approach for generati...
research
05/08/2022

On Conditioning the Input Noise for Controlled Image Generation with Diffusion Models

Conditional image generation has paved the way for several breakthroughs...
research
07/04/2023

DiT-3D: Exploring Plain Diffusion Transformers for 3D Shape Generation

Recent Diffusion Transformers (e.g., DiT) have demonstrated their powerf...
research
09/01/2023

Iterative Multi-granular Image Editing using Diffusion Models

Recent advances in text-guided image synthesis has dramatically changed ...
research
12/21/2022

Hierarchically branched diffusion models for efficient and interpretable multi-class conditional generation

Diffusion models have achieved justifiable popularity by attaining state...
research
03/04/2021

Perceiver: General Perception with Iterative Attention

Biological systems understand the world by simultaneously processing hig...

Please sign up or login with your details

Forgot password? Click here to reset