CaDM: Codec-aware Diffusion Modeling for Neural-enhanced Video Streaming

11/15/2022
by   Qihua Zhou, et al.
18

Recent years have witnessed the dramatic growth of Internet video traffic, where the video bitstreams are often compressed and delivered in low quality to fit the streamer's uplink bandwidth. To alleviate the quality degradation, it comes the rise of Neural-enhanced Video Streaming (NVS), which shows great prospects for recovering low-quality videos by mostly deploying neural super-resolution (SR) on the media server. Despite its benefit, we reveal that current mainstream works with SR enhancement have not achieved the desired rate-distortion trade-off between bitrate saving and quality restoration, due to: (1) overemphasizing the enhancement on the decoder side while omitting the co-design of encoder, (2) limited generative capacity to recover high-fidelity perceptual details, and (3) optimizing the compression-and-restoration pipeline from the resolution perspective solely, without considering color bit-depth. Aiming at overcoming these limitations, we are the first to conduct an encoder-decoder (i.e., codec) synergy by leveraging the inherent visual-generative property of diffusion models. Specifically, we present the Codec-aware Diffusion Modeling (CaDM), a novel NVS paradigm to significantly reduce streaming delivery bitrates while holding pretty higher restoration capacity over existing methods. First, CaDM improves the encoder's compression efficiency by simultaneously reducing resolution and color bit-depth of video frames. Second, CaDM empowers the decoder with high-quality enhancement by making the denoising diffusion restoration aware of encoder's resolution-color conditions. Evaluation on public cloud services with OpenMMLab benchmarks shows that CaDM effectively saves up to 5.12 - 21.44 times bitrates based on common video standards and achieves much better recovery quality (e.g., FID of 0.61) over state-of-the-art neural-enhancing methods.

READ FULL TEXT

page 2

page 4

page 7

research
03/04/2020

Learning for Video Compression with Hierarchical Quality and Recurrent Enhancement

The recent years have witnessed the great potential of deep learning for...
research
07/22/2023

Real-Time Neural Video Recovery and Enhancement on Mobile Devices

As mobile devices become increasingly popular for video streaming, it's ...
research
07/05/2022

Deep Parametric 3D Filters for Joint Video Denoising and Illumination Enhancement in Video Super Resolution

Despite the quality improvement brought by the recent methods, video sup...
research
05/08/2023

Compressed Video Quality Assessment for Super-Resolution: a Benchmark and a Quality Metric

We developed a super-resolution (SR) benchmark to analyze SR's capacity ...
research
11/22/2022

SuperTran: Reference Based Video Transformer for Enhancing Low Bitrate Streams in Real Time

This work focuses on low bitrate video streaming scenarios (e.g. 50 - 20...
research
06/07/2021

Deep Neural Network-based Enhancement for Image and Video Streaming Systems: A Survey and Future Directions

Internet-enabled smartphones and ultra-wide displays are transforming a ...
research
10/12/2020

Neural Enhancement in Content Delivery Systems: The State-of-the-Art and Future Directions

Internet-enabled smartphones and ultra-wide displays are transforming a ...

Please sign up or login with your details

Forgot password? Click here to reset