Spatio-Temporal Multi-Flow Network for Video Frame Interpolation

11/30/2021
by   Duolikun Danier, et al.
13

Video frame interpolation (VFI) is currently a very active research topic, with applications spanning computer vision, post production and video encoding. VFI can be extremely challenging, particularly in sequences containing large motions, occlusions or dynamic textures, where existing approaches fail to offer perceptually robust interpolation performance. In this context, we present a novel deep learning based VFI method, ST-MFNet, based on a Spatio-Temporal Multi-Flow architecture. ST-MFNet employs a new multi-scale multi-flow predictor to estimate many-to-one intermediate flows, which are combined with conventional one-to-one optical flows to capture both large and complex motions. In order to enhance interpolation performance for various textures, a 3D CNN is also employed to model the content dynamics over an extended temporal window. Moreover, ST-MFNet has been trained within an ST-GAN framework, which was originally developed for texture synthesis, with the aim of further improving perceptual interpolation quality. Our approach has been comprehensively evaluated – compared with fourteen state-of-the-art VFI algorithms – clearly demonstrating that ST-MFNet consistently outperforms these benchmarks on varied and representative test datasets, with significant gains up to 1.09dB in PSNR for cases including large motions and dynamic textures. Project page: https://danielism97.github.io/ST-MFNet.

READ FULL TEXT

page 1

page 4

page 6

page 8

page 10

research
02/15/2022

Enhancing Deformable Convolution based Video Frame Interpolation with Coarse-to-fine 3D CNN

This paper presents a new deformable convolution-based video frame inter...
research
02/26/2021

Texture-aware Video Frame Interpolation

Temporal interpolation has the potential to be a powerful tool for video...
research
03/27/2023

NeuralPCI: Spatio-temporal Neural Field for 3D Point Cloud Multi-frame Non-linear Interpolation

In recent years, there has been a significant increase in focus on the i...
research
06/14/2021

TimeLens: Event-based Video Frame Interpolation

State-of-the-art frame interpolation methods generate intermediate frame...
research
03/16/2023

LDMVFI: Video Frame Interpolation with Latent Diffusion Models

Existing works on video frame interpolation (VFI) mostly employ deep neu...
research
04/06/2021

Deep Animation Video Interpolation in the Wild

In the animation industry, cartoon videos are usually produced at low fr...
research
09/27/2022

Frame Interpolation for Dynamic Scenes with Implicit Flow Encoding

In this paper, we propose an algorithm to interpolate between a pair of ...

Please sign up or login with your details

Forgot password? Click here to reset