Beyond a Video Frame Interpolator: A Space Decoupled Learning Approach to Continuous Image Transition

03/18/2022
by   Tao Yang, et al.
19

Video frame interpolation (VFI) aims to improve the temporal resolution of a video sequence. Most of the existing deep learning based VFI methods adopt off-the-shelf optical flow algorithms to estimate the bidirectional flows and interpolate the missing frames accordingly. Though having achieved a great success, these methods require much human experience to tune the bidirectional flows and often generate unpleasant results when the estimated flows are not accurate. In this work, we rethink the VFI problem and formulate it as a continuous image transition (CIT) task, whose key issue is to transition an image from one space to another space continuously. More specifically, we learn to implicitly decouple the images into a translatable flow space and a non-translatable feature space. The former depicts the translatable states between the given images, while the later aims to reconstruct the intermediate features that cannot be directly translated. In this way, we can easily perform image interpolation in the flow space and intermediate image synthesis in the feature space, obtaining a CIT model. The proposed space decoupled learning (SDL) approach is simple to implement, while it provides an effective framework to a variety of CIT problems beyond VFI, such as style transfer and image morphing. Our extensive experiments on a variety of CIT tasks demonstrate the superiority of SDL to existing methods. The source code and models can be found at <https://github.com/yangxy/SDL>.

READ FULL TEXT

page 8

page 11

page 12

page 13

page 14

research
11/12/2020

RIFE: Real-Time Intermediate Flow Estimation for Video Frame Interpolation

We propose RIFE, a Real-time Intermediate Flow Estimation algorithm for ...
research
09/27/2022

Frame Interpolation for Dynamic Scenes with Implicit Flow Encoding

In this paper, we propose an algorithm to interpolate between a pair of ...
research
04/15/2021

Zooming SlowMo: An Efficient One-Stage Framework for Space-Time Video Super-Resolution

In this paper, we address the space-time video super-resolution, which a...
research
11/30/2017

Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation

Given two consecutive frames, video interpolation aims at generating int...
research
01/25/2022

Splatting-based Synthesis for Video Frame Interpolation

Frame interpolation is an essential video processing technique that adju...
research
11/26/2018

Deep Network Interpolation for Continuous Imagery Effect Transition

Deep convolutional neural network has demonstrated its capability of lea...
research
03/29/2021

AlignMix: Improving representation by interpolating aligned features

Mixup is a powerful data augmentation method that interpolates between t...

Please sign up or login with your details

Forgot password? Click here to reset