Diagnosing and Preventing Instabilities in Recurrent Video Processing

10/10/2020
by   Thomas Tanay, et al.
0

Recurrent models are becoming a popular choice for video enhancement tasks such as video denoising. In this work, we focus on their stability as dynamical systems and show that they tend to fail catastrophically at inference time on long video sequences. To address this issue, we (1) introduce a diagnostic tool which produces adversarial input sequences optimized to trigger instabilities and that can be interpreted as visualizations of spatio-temporal receptive fields, and (2) propose two approaches to enforce the stability of a model: constraining the spectral norm or constraining the stable rank of its convolutional layers. We then introduce Stable Rank Normalization of the Layers (SRNL), a new algorithm that enforces these constraints, and verify experimentally that it successfully results in stable recurrent video processing.

READ FULL TEXT

page 2

page 5

page 8

page 9

page 12

page 13

page 14

page 15

research
12/16/2021

Stable Long-Term Recurrent Video Super-Resolution

Recurrent models have gained popularity in deep learning (DL) based vide...
research
11/17/2014

Long-term Recurrent Convolutional Networks for Visual Recognition and Description

Models based on deep convolutional networks have dominated recent image ...
research
12/24/2020

An Efficient Recurrent Adversarial Framework for Unsupervised Real-Time Video Enhancement

Video enhancement is a challenging problem, more than that of stills, ma...
research
06/06/2020

Learning Memory-Efficient Stable Linear Dynamical Systems for Prediction and Control

Learning a stable Linear Dynamical System (LDS) from data involves creat...
research
05/24/2017

Adaptive Detrending to Accelerate Convolutional Gated Recurrent Unit Training for Contextual Video Recognition

Based on the progress of image recognition, video recognition has been e...
research
04/19/2018

NAIS-Net: Stable Deep Networks from Non-Autonomous Differential Equations

This paper introduces "Non-Autonomous Input-Output Stable Network" (NAIS...
research
08/07/2023

Recurrent Self-Supervised Video Denoising with Denser Receptive Field

Self-supervised video denoising has seen decent progress through the use...

Please sign up or login with your details

Forgot password? Click here to reset