Beyond Natural Motion: Exploring Discontinuity for Video Frame Interpolation

02/15/2022
by   Sangjin Lee, et al.
8

Video interpolation is the task that synthesizes the intermediate frame given two consecutive frames. Most of the previous studies have focused on appropriate frame warping operations and refinement modules for the warped frames. These studies have been conducted on natural videos having only continuous motions. However, many practical videos contain a lot of discontinuous motions, such as chat windows, watermarks, GUI elements, or subtitles. We propose three techniques to expand the concept of transition between two consecutive frames to address these issues. First is a new architecture that can separate continuous and discontinuous motion areas. We also propose a novel data augmentation strategy called figure-text mixing (FTM) to make our model learn more general scenarios. Finally, we propose loss functions to give supervisions of the discontinuous motion areas with the data augmentation. We collected a special dataset consisting of some mobile games and chatting videos. We show that our method significantly improves the interpolation qualities of the videos on the special dataset. Moreover, our model outperforms the state-of-the-art methods for natural video datasets containing only continuous motions, such as DAVIS and UCF101.

READ FULL TEXT

page 3

page 5

page 6

page 7

page 8

research
06/25/2022

Enhanced Deep Animation Video Interpolation

Existing learning-based frame interpolation algorithms extract consecuti...
research
03/19/2020

Photo-Realistic Video Prediction on Natural Videos of Largely Changing Frames

Recent advances in deep learning have significantly improved performance...
research
06/24/2023

Boost Video Frame Interpolation via Motion Adaptation

Video frame interpolation (VFI) is a challenging task that aims to gener...
research
11/21/2022

H-VFI: Hierarchical Frame Interpolation for Videos with Large Motions

Capitalizing on the rapid development of neural networks, recent video f...
research
04/06/2021

Deep Animation Video Interpolation in the Wild

In the animation industry, cartoon videos are usually produced at low fr...
research
06/10/2023

Shuffled Autoregression For Motion Interpolation

This work aims to provide a deep-learning solution for the motion interp...
research
03/27/2023

Continuous Intermediate Token Learning with Implicit Motion Manifold for Keyframe Based Motion Interpolation

Deriving sophisticated 3D motions from sparse keyframes is a particularl...

Please sign up or login with your details

Forgot password? Click here to reset