Neural Video Coding using Multiscale Motion Compensation and Spatiotemporal Context Model

by   Haojie Liu, et al.

Over the past two decades, traditional block-based video coding has made remarkable progress and spawned a series of well-known standards such as MPEG-4, H.264/AVC and H.265/HEVC. On the other hand, deep neural networks (DNNs) have shown their powerful capacity for visual content understanding, feature extraction and compact representation. Some previous works have explored the learnt video coding algorithms in an end-to-end manner, which show the great potential compared with traditional methods. In this paper, we propose an end-to-end deep neural video coding framework (NVC), which uses variational autoencoders (VAEs) with joint spatial and temporal prior aggregation (PA) to exploit the correlations in intra-frame pixels, inter-frame motions and inter-frame compensation residuals, respectively. Novel features of NVC include: 1) To estimate and compensate motion over a large range of magnitudes, we propose an unsupervised multiscale motion compensation network (MS-MCN) together with a pyramid decoder in the VAE for coding motion features that generates multiscale flow fields, 2) we design a novel adaptive spatiotemporal context model for efficient entropy coding for motion information, 3) we adopt nonlocal attention modules (NLAM) at the bottlenecks of the VAEs for implicit adaptive feature extraction and activation, leveraging its high transformation capacity and unequal weighting with joint global and local information, and 4) we introduce multi-module optimization and a multi-frame training strategy to minimize the temporal error propagation among P-frames. NVC is evaluated for the low-delay causal settings and compared with H.265/HEVC, H.264/AVC and the other learnt video compression methods following the common test conditions, demonstrating consistent gains across all popular test sequences for both PSNR and MS-SSIM distortion metrics.


page 1

page 3

page 7

page 12


End-to-end Neural Video Coding Using a Compound Spatiotemporal Representation

Recent years have witnessed rapid advances in learnt video coding. Most ...

Learned Video Compression via Joint Spatial-Temporal Correlation Exploration

Traditional video compression technologies have been developed over deca...

DVMark: A Deep Multiscale Framework for Video Watermarking

Video watermarking embeds a message into a cover video in an imperceptib...

Adaptation and Attention for Neural Video Coding

Neural image coding represents now the state-of-the-art image compressio...

Spatiotemporal Entropy Model is All You Need for Learned Video Compression

The framework of dominant learned video compression methods is usually c...

End-to-End Learning for Video Frame Compression with Self-Attention

One of the core components of conventional (i.e., non-learned) video cod...

Deep Video Coding with Dual-Path Generative Adversarial Network

The deep-learning-based video coding has attracted substantial attention...

I Introduction

Compressed video, a dominant media representation across the entire Internet, occupies more than 70% total traffic volume nowadays for entertainment (e.g., YouTube), productivity (e.g., tele-education), security (e.g., surveillance) etc. It still keeps growing explosively. Thus, in pursuit of efficient storage and network transmission, and pristine quality of experience (QoE) with higher resolution content (e.g., 2K, 4K and even 8K video with frame rate at 30 Hz, 60 Hz or even more), a better compression approach is greatly and continuously desired. In principle, the key problem in video coding is how to efficiently exploit visual signal redundancy using prior information, spatially (e.g., intra prediction, transform), temporally (e.g., inter prediction), and statistically (e.g., entropy context adaptation) for more compact representations with less bit rate consumption at the same reconstruction quality. This is well formulated as the minimization of Lagrangian cost of rate-distortion optimization (RDO) that is widely adopted in existing video coders, e.g.,


with and represent the compressed bit rates and reconstructed distortion respectively.

Motivation. Over the past three decades, video compression technologies have been evolving and adapting constantly with coding efficiency improved by several folds, mostly driven under the efforts from the experts in ISO/IEC Moving Picture Experts Group (MPEG), ITU-T Video Coding Experts Group (VCEG) and their joint task forces. It leads to several popular video coding standards, including the H.264/Advanced Video Coding (H.264/AVC) [46], High-Efficiency Video Coding (HEVC) [41] and emerging versatile video coding (VVC) [6]. These standards share the similar (recursive) block-based hybrid prediction/transform framework where individual coding tools, such as the intra/inter prediction, integer transforms, context-adaptive entropy coding, etc, are intensively handcrafted to optimize the overall efficiency. Among them, pixel-domain predictive coding is one of the most important factors, contributing to the major performance gains [36]

. For example, pixel-domain intra prediction was officially adopted into the H.264/AVC and later extended with the support of recursive block-sizes and abundant predictive directions for efficiently exploiting spatial structures; recursive and even non-squared blocks are extensively used in inter prediction to remove temporal coherency. Basically, conventional video coding methods leverage the spatiotemporal pixel neighbors (as well as their linear combinations) for predictive signal construction, resulting in corresponding residuals for subsequent transform, quantization, and entropy coding for more compact representation. Optimal coding mode with appropriate block size and orientation (e.g., intra direction, inter motion vectors) is selected via computational RDO process, utilizing

-norm (e.g., mean absolute error - MAE) or -norm (e.g., mean squared error - MSE) as the distortion metric.

Though recursive block-based pixel prediction shows its great success, it is mainly due to the hardware advancements in past decades, by which we can exhaustively search for the best prediction. It, however, is more and more challenging to simply trade computational resources for efficiency improvement because Moore’s Law does not hold any more [13]. It therefore calls for innovative methodologies and architectures of video coding to further improve the coding efficiency, in response to the ever increasing users’ requirement for video resolution and quality. Such pixel prediction strategy, either intra or inter, mostly relies on the physical coherence of video signal and applies the mathematical tools (e.g., linear weighting, orthogonal transform, Lagrangian optimization) for signal energy compaction.

Our Approach. We propose an end-to-end neural video coding framework (NVC), which codes intra-frame pixels (called neuro-Intra), inter-frame motion (called neuro-Motion), and inter-frame residual (called neuro-Res) using separate variational autoencoders (VAE), as shown in Fig. 1. A multiscale motion compensation network (MS-MCN) works together with neuro-Motion to generate multiscale optical flows and perform multiscale motion-compensated prediction of the current frame from the previous frame. The sparse image differences between past and present frame, e.g., residuals, are then encoded to obtain the final reconstruction; All three VAEs, e.g., neuro-Intra, neuro-Motion, neuro-Res for compressing intra-pixel, inter-motion and inter-residual, are engineered together with MS-MCN in an end-to-end learning manner. Note that neuro-Intra takes a native image frame as input, neuro-Motion uses the current and past reconstructed frame, generating multiscale compressed flows (MCFs), MS-MCN uses these generated MCFs for motion compensation to obtain the inter-predicted frame and neuro-Res encodes the difference between the current frame and its prediction for the final reconstruction. Additionally, joint spatiotemporal and hyper priors are aggregated for efficient and adaptive context modeling of latent features to improve entropy coding efficiency for the motion field.

We have evaluated the efficiency of the proposed NVC for the low-delay causal settings against well-known HEVC, H.264/AVC and other learnt video compression methods following the common test conditions. The NVC demonstrated the leading performance with consistent gains across all popular test sequences for both PSNR (Peak signal-to-noise ratio) and MS-SSIM (multiscale structural similarity) 

[45] distortion metrics. Using the H.264/AVC as a common anchor, our NVC presents 35% BD-Rate (Bjontegaard Delta Rate) [5] gains, while HEVC and DVC (Deep Video Coding) [31] offer 30% and 22% gains, respectively, when the distortion is measured in terms of PSNR. Gains are even larger, if the distortion metric is replaced by the MS-SSIM. In this case, NVC can achieve 50% improvement, while both HEVC and DVC are around 25%. We further compare our NVC with DVC_Pro [32] (an upgraded version of DVC) on HEVC test sequences: our NVC reveals 32.20% and 50.07% BD-Rate reduction measured by PSNR and MS-SSIM distortion respectively, while DVC_Pro gives 34.57% and 45.88%.

Ablation studies have also been conducted to examine the gains due to different modules of NVC. We have shown that temporal priors and multi-frame training could greatly improve the efficiency and learning stability. Our MS-MCN is able to remove motion compensation noise by multiscale compensation, even better than using a cascaded trained denoising network.

abber. description
NLAM NonLocal Attention Module
LAM Local Attention Module
MS-MCN Multi-scale Motion Compensation Network
SS-MCN Single-scale Motion Compensation Network
PA Prior Aggregation
MCF Multiscale Compressed Flow
neuro-Intra Neural intra coding
neuro-Motion Neural motion coding
neuro-Res Neural residual coding
MSE Mean Squared Error
MAE Mean Absolute Error
PSNR Peak signal-to-noise ratio
MS-SSIM Multiscale Structural Similarity
TABLE I: Abbreviations and Notations
Fig. 1: Neural Video Coding (NVC). The modules neuro-Intra, neuro-Res, and neuro-Motion follow the general model architecture in Fig. 2 for efficient representations of intra pixels, displaced inter residuals, and inter motions. The neuro-Motion uses a pyramid decoder for the main decoder as discussed in Sec. III-D2.
Fig. 2: The general structure of the variational autoencoder based compression engine used for neuro-Intra, neuro-Motion and neuro-Res in Fig. 1. The main encoder includes for major convolutional layers (each including a convolutional layer followed by down-sampling and three residual blocks, and the last layer includes a non-local attention module). The hyper encoder includes two major convolutional layers. The main decoder and hyper decoder reverse the processing of and

, respectively. Prior aggregation (PA) engine collects the information from hyper prior, autoregressive spatial neighbors, as well as temporal priors (if applicable) for efficient modeling of the probability distribution of latent features generated by the main encoder. Nonlocal attention is adopted at the bottlenecks of both main and hyper encoders to enable saliency based bit allocation, and rectified linear unit (ReLU) is embedded with convolutions for enabling the nonlinearity. “Q” is for quantization, AE and AD are for arithmetic encoding and decoding, respectively. 2

and 2 are downsampling and upsampling at a factor of 2 for both horizontal and vertical dimensions.

Novelty. The main contributions are highlighted below:

  • We propose an end-to-end deep neural video coding framework (NVC), leveraging learnt feature domain representations for intra-pixel, inter-motion and inter-residual, respectively for compression;

  • neuro-Motion and multiscale motion compensation network (MS-MCN) are employed together to capture coarse-to-fine motion displacements and obtain the prediction by warping the features of the reference frame at multiple scales;

  • We propose a novel spatiotemporal context modeling approach for the entropy coding of the motion features, where the temporal context is obtained through modeling the temporal evolution of the motion features using a ConvLSTM in the temporal updating module (TUM). The temporal context is combined with the autoregressive spatial context and hyperprior features in a spatiotemporal hyper aggregation module (STHAM);

  • Nonlocal attention is attached at bottleneck layers of the VAE modules for adaptive bits allocation based on joint global and local feature extraction implicitly.

This work is based on our preliminary work [30] but with significant extensions and discussions including pyramid flow decoder in neuro-Motion, replacing single scale motion compensation with multiscale motion compensation, spatiotemporal prior aggregation engine for context modeling and progressive training with multiple frames.

The rest of this paper is structured as follows: Sec. II will briefly review relevant studies about leaned image and video coding. Our neural video coding framework (NVC) is given in Sec. III with detailed discussions about neuro-Intra, neuro-Motion, MS-MCN and neuro-Res; Sec. IV will present experiments and ablation studys; and concluding remarks are drawn in Sec. V.

Ii Related Work

Built on advancements of deep neural networks (DNNs), we have seen the explosive growth of DNN-based image/video compression approaches. Some explorations have attempted to replace modular components in traditional image/video coding framework such as filtering, intra prediction, etc; and some others have fully relied on powerful learning tools to perform the end-to-end optimization. Given that our neural video coding framework (NVC) belongs to the end-to-end learning category, we will emphasize the reviews in this avenue. For the modular optimization using DNNs, two great review articles can be found in [28, 33].

Ii-a Learnt Image Compression

DNN-based image compressions usually utilize auto-encoder or VAE architectures, consisting of nonlinear transforms, attention or importance map, differentiable quantization, context model, and embedded loss functions, for end-to-end learning.

Recurrent Autoencoder. Toderic et al. [43] first proposed to utilize fully-connected recurrent auto-encoders for variable-rate thumbnail image compression. A serial improvements were then extended, including the full-resolution image support, learned entropy coding, unequal bits allocation, etc [44, 20] by the introductions of ConvLSTM or ConvGRU, for better coding efficiency. Variable bit rate is intrinsically enabled by such recurrent structure. It, however, suffers from the higher computational complexity at higher bit rates, because more recurrent processing are desired.

Convolutional Autoencoder. Alternatively, convolutional auto-encoders [2, 3, 34, 35], are extensively studied in past years where different bit rates are adapted by setting a variety of in learning to optimize (1). Note that different network models may be required for individual bit rates, making it difficult for hardware implementation (e.g., model adaptation for various bit rates). Recently, conditional convolution [11] and scaling factor [8] were proposed to enable variable-rate compression using a single or a limited number of network parameters without noticeable coding efficiency loss. It makes the convolutional autoencoders more attractive for practical applications.

Attention/Importance Map. Li et al. [25] utilized a separate three-layer CNN to generate importance map for spatial-complexity-based adaptive bits allocation, leading to the noticeable subject quality improvement with well-preserved edges and textures. Instead, Mentzer et al. [34] further selected one channel from the bottleneck layer to unequally weigh features at different spatial locations for simplification. Such importance map embedding was lightweight and easy for training and end-to-end optimization. This approach was later improved with nonlocal attention mechanism to efficiently and implicitly capture both global and local important information for better compression [8].

Nonlinear Transform. To generate more compact feature representation, Balle et al. [2] suggested to replace the traditional nonlinear activation, e.g., ReLU, with the generalized divisive normalization (GDN) that was theoretically proven to be more consistent with image natural statistics for visual perception. A succeeding investigation by the same authors was given in [4], reporting that GDN outperformed other nonlinear rectifiers, such as ReLU, leakyReLU and tanh, in compression tasks. Several follow-up studies [22, 21] directly applied GDN in their networks for compression exploration.

Adaptive Contexts.

Probabilistic model plays a vital role in data compression. Assuming the Gaussian distribution for feature elements, Balle et al. 

[3] utilized hyper priors to estimate the parameters of Gaussian Scale Model (GSM) for latent features. Later Hu et al. [16] used hierarchical hyper priors (coarse-to-fine) for improving the entropy models in multiscale representations. Minnen et al. [35]

improved the context modeling using joint autoregressive spatial neighbors and hyper priors based on Gaussian Mixture Model (GMM). Autoregressive spatial priors were usually extracted by PixelCNNs or PixelRNNs 

[37], which have been widely adopted for natural image density modeling. Reed et al. [38] further introduced multiscale PixelCNNs, yielding competitive density estimation and great speedup (e.g., from to ). It was later extended from 2D architectures to 3D PixelCNNs [34]. Channel-wise weights sharing-based 3D implementations can greatly reduce network parameters with higher parallelization. Then Chen et al. [8] discussed parallel pipelines of 3D PixelCNNs for practical decoding. Previous methods accumulated all the accessible priors to estimate the probability based on a single Gaussian distribution for each element. Recent explorations have shown that weighted GMMs can further improvre the coding efficiency as reported by [10, 23].

Quantization. Quantization is a non-differentiable operation, basically converting continuous variables into discrete variables with a limited alphabet. This process has to be replaced by a differentiable operation when used in end-to-end learning framework for back propagation. A number of methods, such as uniform noise adding [2], stochastic rounding [43], soft-to-hard vector quantization [34] and universal quantization [11], were developed to approximate a continuous distribution for differentiation.

Loss Functions. Pixel-error, such as MSE, or MAE, was one of the most popular loss functions used. In the meantime, SSIM or MS-SSIM was also adopted because of its better consistency with visual perception. Simulations had revealed that SSIM-based loss can improve the perception quality, especially at low bit rates. Towards the perception-optimized encoding, perceptual losses that were measured by adversarial loss [39, 17, 1] and VGG loss [29] were embedded in learning to produce visually appealing results.

Ii-B Learnt Video Compression

Learnt video compression is extended from the learnt image compression by further exploiting the temporal redundancy in a trainable way for efficient representations of temporal motion and displaced image difference (e.g., predictive residual). In most cases, network models for image compression were directly re-used for temporal displaced residuals, leaving a great deal of efforts devoted for better motion field compression.

Chen et al. [9] developed the DeepCoder where a simple convolutional autoencoder was applied for both intra and residual coding at fixed 3232 blocks, and block-based motion estimation in traditional video coding was re-used for temporal compensation. Lu et al.[31] introduced the optical flow for motion representation in their DVC work, which, together with the intra coding in [3], demonstrated similar performance compared with the HEVC. However, the coding efficiency suffer from a sharp loss at low bit rates. Liu et al. [30] extended their nonlocal attention optimized image compression (NLAIC) for coding the intra and residual frame, and applied spatiotempoal adaptive context model for more compact motion representation, showing consistent rate-distortion gains across different contents and bits rates.

Motion can be also implicitly inferred by temporal interpolations. For example, Wu et al. 


applied recurrent neural network (RNN)-based frame interpolation. Together with the residual compensation, it offered comparable performance with H.264/AVC. Djelouah et al. 

[14] further imporved the interpolation-based video coding by utilizing the advanced optical flow estimation and feature domain residual coding. Yang et al. [49] also use a interpolation method with three hierarchical quality layers and a recurrent enhancement network for compression. However, temporal interpolation usually led to coding delay that may not be acceptable for low-latency applications.

Another interesting exploration made by Ripple et al. in [40] was to jointly encode the flow and residual signals using unified quantized features in an unsupervised way. A recurrent state was embedded to aggregate multi-frame information for efficient flow generation and residual coding.

Iii Neural Video Coding

Iii-a Overview of NVC

Our NVC framework is designed for low-delay applications. As with all modern video encoders, the proposed NVC compresses the first frame in each group of pictures as an intra-frame using a VAE based compression engine (neuro-Intra). It codes remaining frames using motion compensated prediction. As shown in Fig. 1, it uses the VAE compressor (neuro-Motion) to generate the multiscale motion field between the current frame and the reference frame. Then, MS-MCN takes multiscale compressed flows, warps the multiscale features of the reference frame, and combines these warped features to generate the predicted frame. The prediction residual is then coded using another VAE-based compressor (neuro-Res).

Given a group of pictures (GOP) = {}, we first encode using the neuro-Intra module, leading to the reconstructed frame . The following frame is encoded predictively, using neuro-Motion, MS-MCN, and neuro-Res together, shown in Fig. 1. Note that MS-MCN takes the multiscale optical flows derived by the pyramid decoder in neuro-Motion, and then generates the predicted frame by multiscale motion compensation. Displaced inter-residual is then compressed in neuro-Res, yielding the reconstruction . The final reconstruction is given by . Encoding of the next P-frame follows the same procedure of until all frames in the GOP are coded completely.

Table I summarizes relevant abbreviations used throughput this paper.

Iii-B The VAE Architecture using NLAM and Spatiotemporal Priors

The general architecture of the VAE model is shown in Fig. 2, with main encoder-decoder pair for latent feature analysis and synthesis, and hyper encoder-decoder for hyper prior generation. The main encoder

uses four stacked CNN layers, where each convolutional layer employs stride convolutions to achieve downsampling (e.g., at a factor of 2 in this example) and cascaded convolutions (e.g., three ResNet-based residual blocks 

[15]111We apply cascaded ResNets for stacked CNNs because of its high-efficiency and reliable performance. Other efficient CNNs architectures can be applied as well.) for efficient feature extraction. We utilize two-layer hyper encoder to further generate the subsequent hyper priors as side information, which are used for the entropy coding for the latent features.

To capture the spatial locality, we apply convolutional layers with limited receptive field (e.g., 33) that are stacked altogether to simulate the layer-wise feature extraction. These same ideas are used in many relevant studies [3, 35]

. We utilize the simplest ReLU as the nonlinear activation function. Other nonlinear activation functions can be used as well, such as the GDN (Generalized Divisive Normalization) in 


Attention mechanism is leveraged to intelligently allocate bit resource (e.g., via unequal feature quantization) for image/video compression [26, 34]. It basically enables more accurate reconstruction of salient areas. We adopt the nonlocal attention module (e.g., NLAM) at the bottleneck layers of both the main encoder and hyper encoder, prior to quantization to include both global and local information for more accurate importance selection. This module is motivated by the behaviour of HVS, where we often promptly scan the entire viewing scene to have the complete understanding of the field of vision, and then fixate to the salient regions.

To enable more accurate conditional probability density modeling for entropy coding of the latent features, we introduce the Prior Aggregation (PA) engine which fuses the inputs from the hyper priors, spatial neighbors and temporal context (if applicable)222Intra and residual coding only use joint spatial and hyper priors without temporal inference.. The more accurate context modeling requires less resource (e.g., bits) for information representation as suggested in information theory [12]. For the sake of simplicity, we assume the latent features (e.g., motion, image pixel, residual) following the Gaussian distribution as in [35, 16]

, and use the PA engine to derive the mean and standard deviation of the distribution for each feature.

Iii-C Neural Intra Coding

Our neuro-Intra is a simplified version of the NLAIC that was originally proposed in [8].

NLAIC: One major distinction of NLAIC from the VAE model using autoregressive spatial context in [35] is the introduction of a nonlocal attention module (NLAM) inspired by [50]. NLAM is used to capture the global and local importance for saliency aggregation, by which we can assign different importance to various spatial-channel elements implicitly. Such nonlocal attention mechanism is inspired by the visual information processing of spatial perception (e.g., coarse global structure plus fine-grain local details). We have found that with the addition of NLAM, we can achieve similar performance as [35] without using GDN nonlinearity.

In addition, we have applied 3D 555 masked CNN333This 555 convolutional kernel shares the same parameters for all channels, offering great model complexity reduction compared with 2D CNN-based solution in [35]. to extract spatial priors, which are fused with hyper priors in PA for entropy context modeling (e.g., bottom part of Fig. 6). Here, we have assumed the single Gaussian distribution for the context modeling of entropy coding. More details of NLAIC can be found in [8]. Note that temporal priors are not adopted for intra-pixel and inter-residual in this paper.

Improvements to NLAIC. Original NLAIC applies multiple NLAMs in both main and hyper coders, leading to excessive memory consumption at a large spatial scale. In NVC, NLAMs are only used at the bottleneck layers for both main and hyper encoder-decoder pairs, achieving adaptive bits allocation implicitly.

To overcome the non-differentiability of the quantization operation, quantization is simulated by adding uniform noise in [3]. However, such noise augmentation is not exactly consistent with the rounding in inference, yielding the performance loss as reported by [11]. Thus, we apply the universal quantization (UQ) [11] in neuro-Intra, i.e.,:


where is a quantized symbol and represents the random uniform variable ranging from to . Statistically, we define the gradient as 1 for back propagation since UQ can be approximated as a linear function. Such UQ (2) is used for neuro-Motion and neuro-Res as well. When applying to common Kodak dataset, neuro-Intra achieved similar performance as NLAIC [8], outperforming Minnen (2018) [35], BPG (4:4:4) and JPEG2000, as shown in Fig. 3.

Fig. 3: Efficiency of neuro-Intra. PSNR vs. rate performance of neuro-Intra in comparison to NLAIC [8], Minnen (2018) [35], BPG (4:4:4) and JPEG2000. Note that the curves for neuro-Intra and NLAIC overlap.
Fig. 4: Motion Estimation and Compensation. (a) Two-stage single-scale motion coding and compensation approach using a pre-trained flow network (with explicit raw flow) and a cascaded flow compression network (e.g., [31]); (b) One-stage unsupervised motion generation and compensation approach with implicit flow represented by quantized features that will be decoded into a single scale motion field for motion compensation (warping) (e.g., [30]); (c) One-stage neuro-Motion with MS-MCN uses a pyramidal flow decoder to synthesize the multiscale compressed flows (MCFs) that are used in a multiscale motion compensation network for generating predicted frames.

Iii-D Neural Motion Coding and Compensation

Inter-frame coding plays a vital role in video coding. The key is how to efficiently represent motion in a compact format for effective compensation. In comparison to the pixel-domain block-based motion estimation and compensation in conventional video coding, we rely on the optical flow to accurately capture the temporal information for motion compensation.

Iii-D1 Single-scale Motion Generation and Compensation

Deep learning has spawned several optical flow estimation methods in either supervised or unsupervised way, such as FlowNet2 [18], PWC-net [42], etc. They mostly focus on acquiring high-precision optical flow between two consecutive (uncompressed) frames without any compression rate constraint. In video coder, it, however, is more challenging to derive robust flow where bitrate is often limited, and the reference frame is usually lossy encoded with inherent compression noises.

One approach applies a two-stage method shown in Fig. (a)a, which is utilized by DVC [31]. Such two-stage scheme first uses a pre-trained lossless flow generator (e.g., FlowNet2) for explicit flow derivation, and then cascades an auto-encoder to encode the flow. RDO is examined when compressing the optical flow to balance the bits allocation and reconstruction distortion. Either separable or joint optimization of flow derivation and compression can be applied for such two-stage approach.

We propose an alternative one-stage framework, shown in Fig. (b)b, which is first presented in [30]. It directly transforms concatenated two frames (e.g., one is the reference from past, and one is current frame) into quantized temporal features that represent the inter-frame motion. These quantized features are decoded into compressed optical flow in an unsupervised way for frame compensation via warping. Such one-stage scheme does not require any pre-trained flow network such as FlowNet2 or PWC-net to generate the optical flow explicitly. It allows us to quantize the motion features rather than the optical flows and train the motion feature encoder and decoder together with explicit consideration of quantization and rate constraint. Note that the motion features are generated from the main motion encoder, and hyper encoder is used to generate hyper features for motion features in the VAE model in Fig. 2.

Loop filters, or quality enhancement networks, can be used in both two-stage and one-stage methods to enhance the quality of flow-warped frame, for improved quality of predicted frame, as shown in Fig. (a)a and (b)b. This is mainly because linear interpolation-based backward warping would inevitably introduce artifacts in reconstruction, especially for cases having flow estimation errors [19]. A cascaded quality enhancement model can be devised to alleviate such artifacts and improve the reconstruction quality. End-to-end learning can be further applied to the entire framework.

Iii-D2 Multiscale Motion Generation and Compensation

We further extend our earlier work [30] to multiscale motion generation and compensation, for better inter-frame prediction. The neuro-Motion is modified for multiscale motion generation, where the main encoder is used for feature fusion, but we replace the main decoder by a pyramidal flow decoder, which generates the Multiscale Compressed optical Flows (MCFs). MCFs will be processed together with the reference frame, using a multiscale motion compensation network (MS-MCN) to obtain the predicted frame efficiently, as shown in Fig. (c)c. The overall pipeline has four key steps:

  • Step 0: Pretrain the neuro-Motion using loss for MCFs generation. This part involves the implicit motion feature derivation in encoder, and multiscale (compressed) flow decoding in decoder.

    For the former part, we use the main encoder of the VAE (see Fig. 2) to extract temporal features for implicit motion representation. It inputs the concatenation of reference frame and current frame to derive latent motion features. To ensure the accurate MCFs generation, we replace the residual blocks in both VAE encoder and decoder with ResNet-based local attention modules (LAMs)444Local attention module only utilizes local features to generate the attention maps compared with NLAM.. Here, quantized features have a size of with denoting respective height and width of the original frame.

    For the pyramidal flow decoder, we plug additional convolutional layers at each scale after the LAM to produce resolution-dependent flows to capture the multiscale motion fields. The pyramid flow decoder is first trained in an unsupervised way using the loss:


    with as loss. s are scale-dependent weighting coefficients which is set as empirically. Here, refers to the finest resoltion. is obtained by backward warping in each scale:


    Multiscale labels , and multiscale references are generated from the current and reference frames respectively using average pooling with stride 2 for both vertical and horizontal dimensions.

    Fig. 5: Multiscale Motion Compensation. (a) A pyramidal decomposition is applied on reference frame to generate multi-scale features. It consecutively uses a 33 convolution, ReLU activation and a 55 convolution with stride 2. (b) Feature Aggregation between consecutive scales. Each upsampling layer consists of a 55 convolution with stride 2, ReLU activation and a 33 convolution.
  • Step 1: Fine-tune neuro-Motion via rate-constrained loss. We further fine-tune the neuro-Motion with rate constraints, where the bit rate of quantized feature representations will be estimated based on adaptive contexts generated from the hyper features and other spatiotemporal context (to be discussed in Fig. 6).We use a rate-distortion loss as in (1) where the distortion is the multiscale prediction loss as in (3) and the rate is the estimated entropy of the motion features.

  • Step 2: Pretrain MS-MCN via MCFs. Using the MCFs (e.g., s) generated by the neuro-Motion from Step 1, we will pretrain the multiscale motion compensation network for motion compensated prediction in the feature domain. It first uses a pyramidal decomposition of the reference frame, shown in Fig. (a)a, to generate multiscale features of the reference frames which are respectively warped with the MCFs at corresponding scale for progressive aggregation.

    Fig. (b)b exemplifies the aggregation at the smallest with and the second smallest scale with : we use to warp corresponding using (4), leading to the warped representation . This is then upsampled to that has the same resolution as the next scale, and concatenated with . These concatenated features are then upsampled again to yield scale 2 features:


    Note that will be concatenated with using similar steps. The upsampling operator is implemented with two convolution layers as detailed in Fig. (b)b. Eventually, we will apply a fusion layer, consisting of a 11 convolution, ReLU and a 33 convolution, on to obtain the final predicted frame. Pretraining the MS-MCN is accomplished by minimizing the multiscale motion compensation loss in (3).

  • Step 3: End-to-end Joint Refinement. In the end, we take pre-trained neuro-Motion and MS-MCN together and perform an end-to-end joint refinement. We use the rate-constrained loss as in Step 1, and use the reconstructed intra frame as the reference frame. We resort to the “pre-training and joint-refinement” strategy since applying the joint training directly makes the model extremely unstable and hard to converge as revealed experimentally.

Sequences High Bit Rate Low Bit Rate
BasketballPass 32.94 29.34 30.52 30.01
RaceHorses 27.48 25.08 26.12 24.41
PartyScene 27.01 26.43 25.55 25.27
BQMall 33.09 31.35 30.76 30.33
vidyo1 38.81 37.14 36.34 36.30
vidyo4 38.86 36.60 36.40 36.33
Average 33.03 30.99 30.94 30.44
TABLE II: Comparative Studies of MS-MCN and SS-MCN Using the PSNRs of Predicted Frame.

Table II reports the consistent PSNR gains offered by the MS-MCN in Sec. III-D2, in comparison to the SS-MCN [30] in Sec. III-D1, revealing the efficiency of MS-MCN for accurately motion compensated prediction. At high bit rate scenario, almost 2 dB gain is observed; MS-MCN keeps superior efficiency even with challenging temporal motions or occlusions in “BasketballPass” content. At low bit rates, gains are reduced but still noticeable. It shows that the PNSR gains depend on how much motion information are actually compressed. The more information (e.g., high bit rate) comes with larger PSNR gains, and vice versa.

Fig. 6: Context-Adaptive Modeling Using Joint Spatio-temporal and Hyper Priors. All priors are fused in PA to provide estimates of the probability distribution parameters.

Iii-D3 Context-Adaptive Flow Coding

It is well recognized that motion fields present high correlations, both spatially and temporally. To exploit such correlation, Ripple et al. [40] proposed to code the flow differences in feature domain. Here, we propose to exploit this correlation in the context modeling for the entropy coding of the motion features. Specifically, we develop a joint spatiotemporal and hyper prior-based context-adaptive model shown in Fig. 6. This is implemented in PA engine of Fig. 2 for neuro-Motion.

The proposed PA engine for neuro-Motion consists of a spatio-temporal-hyper aggregation module (STHAM) and a temporal updating module (TUM), shown in Fig. 6. At timestamp , STHAM accumulates all the accessible priors and estimate the mean and standard deviation of the assumed Gaussian distribution for each new quantized motion feature using:


Here, are elements of quantized latent motion features for the current frame, is consists of temporal priors derived from the motion features preceding the current frame. includes hyper priors of the quantized motion features. These features are concatenated and fused using stacked 1x1x1 convolutions. Note that masked convolution is used for the spatial features .

To generate the temporal priors, TUM is applied to current quantized features recurrently using a standard ConvLSTM:


where are updated temporal priors for the next frame, is a memory state to control information flow across multiple time instances (e.g., frames). Note that the STHAM and TUM components are trained with other components in neuro-Motion in Step 1 and Step 3 described in Sec. III-D2.

It is worth to point out that leveraging temporal correlation for compact motion representation is also widely explored in traditional video coding approaches. For example, motion vector predictions from spatial and temporal co-located neighbors are standardized in HEVC, by which only motion vector differences (after prediction) are encoded. Here, instead of coding the flow feature differences, we use the flow features in the past to help estimating the probability distribution of the flow features in the current frame.

Iii-E Neural Residual Coding

Inter-frame residual coding is another significant module contributing to the overall system efficiency. It is used to compress the temporal prediction error pixels. It affects the efficiency for next frame prediction since errors usually propagate temporally.

Here we use the VAE architecture in Fig. 2 for encoding the residual . The rate-constrained loss is used:


where is the loss between a residual compensated frame and . neuro-Res will be first pretrained using the predicted frames by the pretrained neuro-Motion and MS-MCN, and a loss function in (8) where the rate only consider the bits for residual. Then we refine it jointly with neuro-Motion and MS-MCN, using a loss where considers the bits for both motion and residual with two frames.

Iii-F Training Strategy

Progressive Training. Training all NVC components directly is difficult and unreliable, because these modules are interdependent with each other. For example, inter prediction depends on the former reconstructed frame and meanwhile a better predicted frame will reduce the residual energy. In our model development, we first train the neuro-Intra for multiple bit rates, using multiple values; Then we pretrain and fine-tune the neuro-Motion and MS-MCN through Step 1-3 described in Sec. III-D2, to minimize a training loss that consider both the prediction error and the rate for motion features. We also pretrain neuro-Res using the predicted frames generated by the trained neuro-Motion and MS-MCN so far, with a loss that considers both bit rate of residual and the final reconstruction of the current frame. Finally, we further refine all the modules for inter-coding, including neuro-Motion, MS-MCN and neoro-Res, using a training loss including the reconstruction error and the total rate for the motion features and residual features. For each target bit rate, we set the for training the inter-coding model to be proportional to the used for training the neuro-Intra module.

Training with Multi-frame Loss. One way to train the inter-coding model (including neuro-Motion, MS-MCN and neuro-Res modules) is to use only two frames as a training sample and use the neuro-Intra module to code the first frame, and use the inter-coding model to code the second frame. This training approach can only learn the intra-to-inter variations, yielding instability and poor reconstruction for the succeeding inter frames distant from the first intra frame. To overcome the quality degradation in future inter-frames, we adopt a multi-frame training strategy.

We first pretrain the inter-coding model using pairs of two successive frames as training samples, where the first frame is encoded and decoded using the neuro-Intra. This step follows the progressive training procedure. We then refine the inter-coding model by using groups of four successive frames as training samples, where the first frame is encoded and decoded by a pretrained neuro-Intra, and each following frame is coded using the inter-coding model and the decoded frame is then used recursively as the reference frame for coding the next frame. We use a loss function that considers the sum of the distortions (in MSE or negative MS-SSIM) in all three P-frames, and the sum of the rate for these frames, so that the model update considers the impact of the propagation of P-frame reconstruction errors. Note that once the inter-coding model is trained, it is applied to all inter frames in a GOP during testing. Although it is possible to improve the performance by training a different inter-coding model for each possible distance between a P-frame and the intra-frame, we choose to train a single model that is used repeatedly for all P-frames, to reduce the implementation complexity.

Iv Experimental Studies

Fig. 7: PSNR vs. Rate Performance.

Iv-a Datasets and Hyperparameters for Model Training

Datasets. The neuro-Intra is trained on COCO [27] and CLIC [7] with training samples randomly cropped into 2562563. The neuro-Motion, MS-MCN and neuro-Res are joint trained using Vimeo 90k [48] with sample size of 1921923.

Our NVC is evaluated using the standard HEVC test sequences and ultra video group (UVG) dataset. HEVC test sequences include different classes, covering the contents with a variety of motion, frame rate, resolution and texture; UVG dataset has seven 1080p videos which are often selected for testing video applications.

Loss Function. We have used either MSE or negative MS-SSIM loss for training the intra-coding and inter-coding modules. For pretraining the neuro-Motion and MS-MCN modules, we use the loss on the prediction error instead which is similar to the mean absolute error (MAE) used in traditional motion estimation. We do not use MS-SSIM loss on the predicted frame because MS-SSIM mainly cares about the structure similarity and may ignore the background noise to some extent. Through experiments, we have found that using MS-SSIM loss for neuro-Motion can lead to temporal error accumulation, which not only increases the bits consumption but also potentially leads to unreliable training.

Hyperparameters and Platform.

The initial learning rate (LR) is set to 10e-4 and is halved for every 10 epochs, and final models are obtained using a LR of 10e-5 for both pretraining and overall training. We apply the distributed training on 4 GPUs (Titan Xp) for 3 days for each bit rate model.

Evaluation Criteria. We apply the same low-delay coding setting as DVC in [31] for our method and traditional H.264/AVC, and HEVC for fair comparison. We encode 100 frames and use GOP of 10 on HEVC test sequences, and 600 frames with GOP of 12 on UVG dataset. Both PSNR and MS-SSIM results are offered to understand the efficiency of NVC.

Generate Models for Different Target Rates. To generate a model for a particular bit rate, we first train the neuro-Intra using a certain value, denoted by . Then we train the inter-frame models (neuro-motion, MS-MCN and neuro-Res) using a proportionally reduced value, This simple way of setting the and values yielded good results. We chose not to further optimize for given .

Iv-B Performance Comparison

Fig. 8: MS-SSIM vs. Rate Performance. NVC shows significant gains for all the testing videos. MS-SSIM is usually more correlated with the perceptual quality than PSNR, especially at low bit rates.
Sequences H.265/HEVC DVC NVC
ClassB -32.03% 0.78 -27.67% 0.0046 -27.92% 0.72 -22.56% 0.0049 -45.66% 1.21 -54.90% 0.0114
ClassC -20.88% 0.91 -19.57% 0.0054 -3.53% 0.13 -24.89% 0.0081 -17.82% 0.73 -43.11% 0.0133
ClassD -12.39% 0.57 -9.68% 0.0023 -6.20% 0.26 -22.44% 0.0067 -15.53% 0.70 -43.64% 0.0123
ClassE -36.45% 0.99 -30.82% 0.0018 -35.94% 1.17 -29.08% 0.0027 -49.81% 1.70 -58.63% 0.0048
UVG -48.53% 1.00 -37.5% 0.0056 -37.74% 1.00 -16.46% 0.0032 -48.91% 1.24 -53.87% 0.0100
Average -30.05% 0.85 -25.04% 0.0039 -22.26% 0.65 -23.08% 0.0051 -35.54% 1.11 -50.83% 0.0103
TABLE III: BD-Rate Gains of NVC, HEVC and DVC against the H.264/AVC.
Fig. 9: Visual Comparison. Reconstructed frames of NVC, HEVC and H.264/AVC. NVC has fewer blocky artifacts and visible noise, etc, and provides better quality at lower bit rate.

Rate distortion Performance. We compare the coding perfomance of different methods in Fig. 7 and 8 using respective PSNR and MS-SSIM measures, across HEVC and UVG test sequences. Note that when we report PSNR or MS-SSIM values, the models are trained using PSNR and MS-SSIM, respectively, as the distortion metric. In Table III, by setting the same anchor using H.264/AVC, our NVC presents 35% BD-Rate gains when the distortion is measured by PSNR, while HEVC offers 30% gains. Gains are even larger, if the distortion is measured by the MS-SSIM. In this case, NVC can achieve 50% improvement, while HEVC is around 25%.

Wu et al. [47] proposed a interpolation based video coding framework and could not get better performance than H.265/HEVC. Our NVC performs significantly better than DVC [31], which has 22.26% and 23.08% BD-Rate reductions under PSNR and MS-SSIM metrics, respectively. From Fig. 7 and Fig. 8, DVC [31] has mainly improved the coding efficiency against HEVC at high bit rates. However, DVC is not competitive at low bit rate (e.g., having performance even worse than H.264/AVC at some rate regimes.). We have also observed that DVC’s performance varies for different test sequences. An improved version of DVC, known as DVC_pro, has been recently reported, which has shown the state of the art performance [32] by using [35] for intra and residual coding and tuning. NVC is slightly better that DVC Pro in both PSNR and MS-SSIM, at 34.5% and 45.88%, respectively.

Visual Comparison. We provide the visual quality comparison with H.264/AVC and H.265/HEVC in Fig. 9. Generally, NVC leads to a higher quality reconstruction at slightly lower bit rate. For example, for the “RaceHorse” which has nontranslational motion and complex background, NVC uses 7% less bits for more than 1.5 dB PSNR improvement compared with H.264/AVC. For other cases, our method also shows robust improvement. Traditional codec usually suffers from blocky artifacts and motion-induced noise close to object edges. One can clearly observe block partition boundaries with severe pixel discontinuity in reconstructed frames by H.264/AVC. Our results have much less visible noise and artifacts.

Iv-C Ablation Study

This section examines modular components in NVC to further understand its capacity for application.

Spatiotemporal Context Modeling. To evaluate the gain from using temporal priors generated by ConvLSTM for the entropy coding of the motion features, we have also trained a model when the PA engine in neuro-Motion does not use the temporal priors. Table IV shows that 2% to 7% rate increase are incurred if the temporal priors are not used. This reveals that temporal priors help to make probability prediction more accurate, leading to less bit consumption for compressing the motion features.

Sequences NVC W/O temporal priors Bits saving
ClassB -54.90% -48.77% 6.13%
ClassC -43.11% -39.17% 3.94%
ClassD -43.64% -39.23% 4.41%
ClassE -58.63% -56.27% 2.36%
UVG dataset -53.87% -49.14% 4.73%
Average -50.83% -46.52% 4.31%
TABLE IV: Efficiency of Temporal Priors for Coding Motion Features. The entries in 2nd and 3rd columns are the BD-rate reduction.

Generally, bits consumed by motion information vary across different content and total target rates. More saving is reported for low bit rates, and for motion intensive content. In these cases, motion bits usually occupy a higher percentage of the total rate. For stationary content, such as HEVC Class E, motion bits contribute less percentage, thus the gain from using temporal priors is less significant.

Fig. 10: Comparison with using a Cascaded Denoising Network. Proposed MS-MCN can achieve better rate-distortion performance compared with using a denoising network following single scale motion compensation.
Fig. 11: Benefit from Multi-frame Training. Two-frame training could only learn temporal variations from intra to inter reconstruction; Multi-frame training is applied to capture both intra-to-inter, and inter-to-inter variations, yielding improved stability and performance.

Comparison with Cascaded Denoising Networks. Linear warping often brings artifacts (e.g., ghosting edges), especially when flow estimation is not accurate and occlusion happens. To remove such artifacts, a denoising network is often added after motion compensated warping as shown in Fig. (a)a and (b)b. Our MS-MCN, instead, rely on multiscale motion fields to generate the predicted frame. As a comparison, we also trained a U-net like denoising network on the warped frames based on the single-scale decoded motion field. As shown in Fig. 10, MS-MCN achieves 0.2 to 0.3 dB PSNR gains across a large bit rate ranges over using a cascaded denoising network for various content for overall performance.

Temporal Stability with Multi-frame Training. Compression Errors (e.g., lossy compression noise) often propagate from one frame to another due to predictive coding structure. It is critical to limit the temporal quality degradation during training. To evaluate the gain from multi-frame training, we compare the performance of the NVC model trained by minimizing reconstruction errors for only one P-frame (single frame training) and the model further refined by minimizing the reconsturction loss for multiple consecutive P-frames. We have varied the number of total frames to use while training, and found that using 4 frames reaches a good compromise. As shown in Fig. 11, multi-frame training could well capture temporal quality variations from intra to inter, and from inter to inter, leading to a more generalized model with consistent performance for all the frames in a GOP with slower quality degradation, and improved stability.

V Conclusion

We have developed an end-to-end deep neural video coding framework (NVC) that can compactly represent the intra-pixel, inter-motion and inter-residual information, respectively. We have shown that the pyramid decoder in neuro-Motion and the multiscale motion compensation network (MS-MCN) together can significantly improve inter-frame prediction, compared to the more conventional single scale motion compensation. Furthermore, adding temporal context can lead to more efficient entropy coding of the motion information than using only spatial context and hyper priors. We further demonstrate that using a multi-frame training loss can effectively mitigate the temporal error propagation.

We evaluate the NVC by PSNR and MS-SSIM respectively, and compare its performance both with standard coding methods including H.264/AVC and H.265/HEVC as well as learnt video coders including Wu et al. [47], DVC [31] and DVC_pro [32]. NVC offers consistent and stable gains over existing methods across a variety of contents and bit rates.

The proposed DNN-based model can be further improved by engineering better stacked CNNs instead of current implementation. Context-adaptive probability models could be further improved. For example, recent exploration in [23] has demonstrated that weighted GMM could further improve the entropy coding efficiency. This can be easily incorporated in our PA engine. Reference frame selection is another direction for overall efficiency optimization, by which we can embed and aggregate most appropriate information for improving the inter-coding efficiency [24].

The H.264/AVC, HEVC, and even VVC, are engineering masterpieces for video coding. adaptation, rate control, etc can be definitely borrowed to improve the NVC. Furthermore, how to make NVC practically applicable is also worth for deep investigation.

To benefit the research community, all relevant materials will be made publically available soon at

Vi Acknowledgement

Our gratitude is directed to anonymous reviewers for their suggestions and efforts.


  • [1] E. Agustsson, M. Tschannen, F. Mentzer, R. Timofte, and L. V. Gool (2019) Generative adversarial networks for extreme learned image compression. In

    Proceedings of the IEEE International Conference on Computer Vision

    pp. 221–231. Cited by: §II-A.
  • [2] J. Ballé, V. Laparra, and E. P. Simoncelli (2016) End-to-end optimized image compression. arXiv preprint arXiv:1611.01704. Cited by: §II-A, §II-A, §II-A, §III-B.
  • [3] J. Ballé, D. Minnen, S. Singh, S. J. Hwang, and N. Johnston (2018) Variational image compression with a scale hyperprior. arXiv preprint arXiv:1802.01436. Cited by: §II-A, §II-A, §II-B, §III-B, §III-C.
  • [4] J. Ballé (2018) Efficient nonlinear transforms for lossy image compression. arXiv preprint arXiv:1802.00847. Cited by: §II-A.
  • [5] G. Bjontegaard (2001) Calculation of average psnr differences between rd-curves. VCEG-M33. Cited by: §I.
  • [6] B. Bross, J. Chen, S. Liu, and Y.-K. Wang (2019-10) Versatile video coding (draft 7). Doc. JVET-P2001. Cited by: §I.
  • [7] (Website) External Links: Link Cited by: §IV-A.
  • [8] T. Chen, H. Liu, Z. Ma, Q. Shen, X. Cao, and Y. Wang (2019) Neural image compression via non-local attention optimization and improved context modeling. arXiv preprint arXiv:1910.06244. Cited by: §II-A, §II-A, §II-A, Fig. 3, §III-C, §III-C, §III-C.
  • [9] T. Chen, H. Liu, Q. Shen, T. Yue, X. Cao, and Z. Ma (2017) DeepCoder: a deep neural network based video compression. In Visual Communications and Image Processing (VCIP), 2017 IEEE, pp. 1–4. Cited by: §II-B.
  • [10] Z. Cheng, H. Sun, M. Takeuchi, and J. Katto (2020) Learned image compression with discretized gaussian mixture likelihoods and attention modules. arXiv preprint arXiv:2001.01568. Cited by: §II-A.
  • [11] Y. Choi, M. El-Khamy, and J. Lee (2019) Variable rate deep image compression with a conditional autoencoder. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3146–3154. Cited by: §II-A, §II-A, §III-C.
  • [12] T. M. Cover and J. A. Thomas (2012) Elements of information theory. John Wiley & Sons. Cited by: §III-B.
  • [13] J. Dean (2020) The deep learning revolution and its implications for computer architecture and chip design. In 2020 IEEE International Solid-State Circuits Conference-(ISSCC), pp. 8–14. Cited by: §I.
  • [14] A. Djelouah, J. Campos, S. Schaub-Meyer, and C. Schroers (2019) Neural inter-frame compression for video coding. In Proceedings of the IEEE International Conference on Computer Vision, pp. 6421–6429. Cited by: §II-B.
  • [15] K. He, X. Zhang, S. Ren, and J. Sun (2016) Deep residual learning for image recognition. In

    Proceedings of the IEEE conference on computer vision and pattern recognition

    pp. 770–778. Cited by: §III-B.
  • [16] Y. Hu, W. Yang, and J. Liu (2020) Coarse-to-fine hyper-prior modeling for learned image compression. In Proc. AAAI Conf. Artif. Intell., pp. 1–8. Cited by: §II-A, §III-B.
  • [17] C. Huang, H. Liu, T. Chen, S. Pu, Q. Shen, and Z. Ma (2019) Extreme image compression via multiscale autoencoders with generative adversarial optimization. In Proc. of IEEE VCIP, Cited by: §II-A.
  • [18] E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and T. Brox (2017) Flownet 2.0: evolution of optical flow estimation with deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2462–2470. Cited by: §III-D1.
  • [19] M. Jaderberg, K. Simonyan, A. Zisserman, et al. (2015) Spatial transformer networks. In Advances in neural information processing systems, pp. 2017–2025. Cited by: §III-D1.
  • [20] N. Johnston, D. Vincent, D. Minnen, M. Covell, S. Singh, T. Chinen, S. Jin Hwang, J. Shor, and G. Toderici (2018) Improved lossy image compression with priming and spatially adaptive bit rates for recurrent networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4385–4393. Cited by: §II-A.
  • [21] J. Klopp, Y. F. Wang, S. Chien, and L. Chen (2018) Learning a code-space predictor by exploiting intra-image-dependencies.. In BMVC, pp. 124. Cited by: §II-A.
  • [22] J. Lee, S. Cho, and S. Beack (2018) Context-adaptive entropy model for end-to-end optimized image compression. arXiv preprint arXiv:1809.10452. Cited by: §II-A.
  • [23] J. Lee, S. Cho, and M. Kim (2019) An end-to-end joint learning scheme of image compression and quality enhancement with improved entropy minimization. arXiv, pp. arXiv–1912. Cited by: §II-A, §V.
  • [24] H. Li, B. Li, and J. Xu (2012) Rate-distortion optimized reference picture management for high efficiency video coding. IEEE transactions on Circuits and Systems for Video Technology 22 (12), pp. 1844–1857. Cited by: §V.
  • [25] M. Li, W. Zuo, S. Gu, D. Zhao, and D. Zhang (2017) Learning convolutional networks for content-weighted image compression. arXiv preprint arXiv:1703.10553. Cited by: §II-A.
  • [26] M. Li, W. Zuo, S. Gu, D. Zhao, and D. Zhang (2018) Learning convolutional networks for content-weighted image compression. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3214–3223. Cited by: §III-B.
  • [27] T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick (2014) Microsoft coco: common objects in context. In European conference on computer vision, pp. 740–755. Cited by: §IV-A.
  • [28] D. Liu, Y. Li, J. Lin, H. Li, and F. Wu (2020) Deep learning-based video coding: a review and a case study. ACM Computing Surveys (CSUR) 53 (1), pp. 1–35. Cited by: §II.
  • [29] H. Liu, T. Chen, Q. Shen, T. Yue, and Z. Ma (2018) Deep image compression via end-to-end learning.. In Proceedings of the IEEE International Conference on Computer Vision Workshops, Cited by: §II-A.
  • [30] H. Liu, L. Huang, M. Lu, T. Chen, Z. Ma, et al. (2020) Learned video compression via joint spatial-temporal correlation exploration. In AAAI, Cited by: §I, §II-B, Fig. 4, §III-D1, §III-D2, §III-D2.
  • [31] G. Lu, W. Ouyang, D. Xu, X. Zhang, C. Cai, and Z. Gao (2019-06) DVC: an end-to-end deep video compression framework. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §I, §II-B, Fig. 4, §III-D1, §IV-A, §IV-B, §V.
  • [32] G. Lu, X. Zhang, W. Ouyang, L. Chen, Z. Gao, and D. Xu (2020) An end-to-end learning framework for video compression. IEEE Transactions on Pattern Analysis and Machine Intelligence. Cited by: §I, §IV-B, §V.
  • [33] S. Ma, X. Zhang, C. Jia, Z. Zhao, S. Wang, and S. Wanga (2019) Image and video compression with neural networks: a review. IEEE Transactions on Circuits and Systems for Video Technology. Cited by: §II.
  • [34] F. Mentzer, E. Agustsson, M. Tschannen, R. Timofte, and L. Van Gool (2018) Conditional probability models for deep image compression. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vol. 1, pp. 3. Cited by: §II-A, §II-A, §II-A, §II-A, §III-B.
  • [35] D. Minnen, J. Ballé, and G. D. Toderici (2018) Joint autoregressive and hierarchical priors for learned image compression. In Advances in Neural Information Processing Systems, pp. 10794–10803. Cited by: §II-A, §II-A, Fig. 3, §III-B, §III-B, §III-C, §III-C, §IV-B, footnote 3.
  • [36] J. Ohm, G. J. Sullivan, H. Schwarz, T. K. Tan, and T. Wiegand (2012) Comparison of the coding efficiency of video coding standards—including high efficiency video coding (hevc). IEEE Transactions on circuits and systems for video technology 22 (12), pp. 1669–1684. Cited by: §I.
  • [37] A. v. d. Oord, N. Kalchbrenner, and K. Kavukcuoglu (2016) Pixel recurrent neural networks. arXiv preprint arXiv:1601.06759. Cited by: §II-A.
  • [38] S. Reed, A. van den Oord, N. Kalchbrenner, S. G. Colmenarejo, Z. Wang, Y. Chen, D. Belov, and N. De Freitas (2017) Parallel multiscale autoregressive density estimation. In

    Proceedings of the 34th International Conference on Machine Learning-Volume 70

    pp. 2912–2921. Cited by: §II-A.
  • [39] O. Rippel and L. Bourdev (2017) Real-time adaptive image compression. arXiv preprint arXiv:1705.05823. Cited by: §II-A.
  • [40] O. Rippel, S. Nair, C. Lew, S. Branson, A. G. Anderson, and L. Bourdev (2019) Learned video compression. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3454–3463. Cited by: §II-B, §III-D3.
  • [41] G. J. Sullivan, J. Ohm, W. Han, and T. Wiegand (2012) Overview of the high efficiency video coding (hevc) standard. IEEE Transactions on circuits and systems for video technology 22 (12), pp. 1649–1668. Cited by: §I.
  • [42] D. Sun, X. Yang, M. Liu, and J. Kautz (2018) Pwc-net: cnns for optical flow using pyramid, warping, and cost volume. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8934–8943. Cited by: §III-D1.
  • [43] G. Toderici, S. M. O’Malley, S. J. Hwang, D. Vincent, D. Minnen, S. Baluja, M. Covell, and R. Sukthankar (2016) Variable rate image compression with recurrent neural networks. In Proceedings of the International Conference on Learning Representations, Cited by: §II-A, §II-A.
  • [44] G. Toderici, D. Vincent, N. Johnston, S. Jin Hwang, D. Minnen, J. Shor, and M. Covell (2017) Full resolution image compression with recurrent neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5306–5314. Cited by: §II-A.
  • [45] Z. Wang, E. P. Simoncelli, and A. C. Bovik (2003) Multiscale structural similarity for image quality assessment. In The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003, Vol. 2, pp. 1398–1402. Cited by: §I.
  • [46] T. Wiegand, G. J. Sullivan, G. Bjontegaard, and A. Luthra (2003) Overview of the h. 264/avc video coding standard. IEEE Transactions on circuits and systems for video technology 13 (7), pp. 560–576. Cited by: §I.
  • [47] C. Wu, N. Singhal, and P. Krähenbühl (2018) Video compression through image interpolation. In ECCV, Cited by: §II-B, §IV-B, §V.
  • [48] T. Xue, B. Chen, J. Wu, D. Wei, and W. T. Freeman (2019) Video enhancement with task-oriented flow. International Journal of Computer Vision 127 (8), pp. 1106–1125. Cited by: §IV-A.
  • [49] R. Yang, F. Mentzer, L. V. Gool, and R. Timofte (2020-06) Learning for video compression with hierarchical quality and recurrent enhancement. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §II-B.
  • [50] Y. Zhang, K. Li, K. Li, B. Zhong, and Y. Fu (2019) Residual non-local attention networks for image restoration. arXiv preprint arXiv:1903.10082. Cited by: §III-C.