Spatially-Adaptive Pixelwise Networks for Fast Image Translation

12/05/2020
by   Tamar Rott Shaham, et al.
0

We introduce a new generator architecture, aimed at fast and efficient high-resolution image-to-image translation. We design the generator to be an extremely lightweight function of the full-resolution image. In fact, we use pixel-wise networks; that is, each pixel is processed independently of others, through a composition of simple affine transformations and nonlinearities. We take three important steps to equip such a seemingly simple function with adequate expressivity. First, the parameters of the pixel-wise networks are spatially varying so they can represent a broader function class than simple 1x1 convolutions. Second, these parameters are predicted by a fast convolutional network that processes an aggressively low-resolution representation of the input; Third, we augment the input image with a sinusoidal encoding of spatial coordinates, which provides an effective inductive bias for generating realistic novel high-frequency image content. As a result, our model is up to 18x faster than state-of-the-art baselines. We achieve this speedup while generating comparable visual quality across different image resolutions and translation domains.

READ FULL TEXT

page 1

page 2

page 4

page 6

page 8

research
08/03/2020

Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation

We present a generic image-to-image translation framework, Pixel2Style2P...
research
10/13/2021

Breaking the Dilemma of Medical Image-to-image Translation

Supervised Pix2Pix and unsupervised Cycle-consistency are two modes that...
research
07/19/2019

Predicting Visual Memory Schemas with Variational Autoencoders

Visual memory schema (VMS) maps show which regions of an image cause tha...
research
08/14/2019

A Tour of Convolutional Networks Guided by Linear Interpreters

Convolutional networks are large linear systems divided into layers and ...
research
12/09/2020

Positional Encoding as Spatial Inductive Bias in GANs

SinGAN shows impressive capability in learning internal patch distributi...
research
09/13/2021

Spatial-Separated Curve Rendering Network for Efficient and High-Resolution Image Harmonization

Image harmonization aims to modify the color of the composited region wi...
research
11/29/2017

Do Convolutional Neural Networks act as Compositional Nearest Neighbors?

We present a simple approach based on pixel-wise nearest neighbors to un...

Please sign up or login with your details

Forgot password? Click here to reset