Temporally Coherent GANs for Video Super-Resolution (TecoGAN)

11/23/2018
by   Mengyu Chu, et al.
7

Adversarial training has been highly successful in the context of image super-resolution. It was demonstrated to yield realistic and highly detailed results. Despite this success, many state-of-the-art methods for video super-resolution still favor simpler norms such as L_2 over adversarial loss functions. This is caused by the fact that the averaging nature of direct vector norms as loss functions leads to temporal smoothness. The lack of spatial detail means temporal coherence is easily established. In our work, we instead propose an adversarial training for video super-resolution that leads to temporally coherent solutions without sacrificing spatial detail. In our generator, we use a recurrent, residual framework that naturally encourages temporal consistency. For adversarial training, we propose a novel spatio-temporal discriminator in combination with motion compensation to guarantee photo-realistic and temporally coherent details in the results. We additionally identify a class of temporal artifacts in these recurrent networks, and propose a novel Ping-Pong loss to remove them. Quantifying the temporal coherence for image super-resolution tasks has also not been addressed previously. We propose a first set of metrics to evaluate the accuracy as well as the perceptual quality of the temporal evolution, and we demonstrate that our method outperforms previous work by yielding realistic and detailed images with natural temporal changes.

READ FULL TEXT

page 1

page 6

page 7

page 8

page 10

page 11

page 16

page 17

research
07/20/2018

Photorealistic Video Super Resolution

With the advent of perceptual loss functions, new possibilities in super...
research
06/13/2020

iSeeBetter: Spatio-temporal video super-resolution using recurrent generative back-projection networks

Recently, learning-based models have enhanced the performance of single-...
research
01/29/2018

tempoGAN: A Temporally Coherent, Volumetric GAN for Super-resolution Fluid Flow

We propose a temporally coherent generative model addressing the super-r...
research
09/07/2021

Perceptual Learned Video Compression with Recurrent Conditional GAN

This paper proposes a Perceptual Learned Video Compression (PLVC) approa...
research
03/16/2019

Robust Super-Resolution GAN, with Manifold-based and Perception Loss

Super-resolution using deep neural networks typically relies on highly c...
research
06/04/2019

A Multi-Pass GAN for Fluid Flow Super-Resolution

We propose a novel method to up-sample volumetric functions with generat...
research
08/17/2020

HRVGAN: High Resolution Video Generation using Spatio-Temporal GAN

In this paper, we present a novel network for high resolution video gene...

Please sign up or login with your details

Forgot password? Click here to reset