End-to-End Learning for Video Frame Compression with Self-Attention

04/20/2020
by   Nannan Zou, et al.
7

One of the core components of conventional (i.e., non-learned) video codecs consists of predicting a frame from a previously-decoded frame, by leveraging temporal correlations. In this paper, we propose an end-to-end learned system for compressing video frames. Instead of relying on pixel-space motion (as with optical flow), our system learns deep embeddings of frames and encodes their difference in latent space. At decoder-side, an attention mechanism is designed to attend to the latent space of frames to decide how different parts of the previous and current frame are combined to form the final predicted current frame. Spatially-varying channel allocation is achieved by using importance masks acting on the feature-channels. The model is trained to reduce the bitrate by minimizing a loss on importance maps and a loss on the probability output by a context model for arithmetic coding. In our experiments, we show that the proposed system achieves high compression rates and high objective visual quality as measured by MS-SSIM and PSNR. Furthermore, we provide ablation studies where we highlight the contribution of different components.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/05/2019

Deep Predictive Video Compression with Bi-directional Prediction

Recently, deep image compression has shown a big progress in terms of co...
research
05/26/2020

End-to-end Optimized Video Compression with MV-Residual Prediction

We present an end-to-end trainable framework for P-frame compression in ...
research
02/11/2021

Frame Difference-Based Temporal Loss for Video Stylization

Neural style transfer models have been used to stylize an ordinary video...
research
11/13/2022

Advancing Learned Video Compression with In-loop Frame Prediction

Recent years have witnessed an increasing interest in end-to-end learned...
research
05/24/2019

From Here to There: Video Inbetweening Using Direct 3D Convolutions

We consider the problem of generating plausible and diverse video sequen...
research
08/06/2020

Optical Flow and Mode Selection for Learning-based Video Coding

This paper introduces a new method for inter-frame coding based on two c...
research
11/16/2018

Learned Video Compression

We present a new algorithm for video coding, learned end-to-end for the ...

Please sign up or login with your details

Forgot password? Click here to reset