DeepAI
Log In Sign Up

Neighbor Correspondence Matching for Flow-based Video Frame Synthesis

07/14/2022
by   Zhaoyang Jia, et al.
0

Video frame synthesis, which consists of interpolation and extrapolation, is an essential video processing technique that can be applied to various scenarios. However, most existing methods cannot handle small objects or large motion well, especially in high-resolution videos such as 4K videos. To eliminate such limitations, we introduce a neighbor correspondence matching (NCM) algorithm for flow-based frame synthesis. Since the current frame is not available in video frame synthesis, NCM is performed in a current-frame-agnostic fashion to establish multi-scale correspondences in the spatial-temporal neighborhoods of each pixel. Based on the powerful motion representation capability of NCM, we further propose to estimate intermediate flows for frame synthesis in a heterogeneous coarse-to-fine scheme. Specifically, the coarse-scale module is designed to leverage neighbor correspondences to capture large motion, while the fine-scale module is more computationally efficient to speed up the estimation process. Both modules are trained progressively to eliminate the resolution gap between training dataset and real-world videos. Experimental results show that NCM achieves state-of-the-art performance on several benchmarks. In addition, NCM can be applied to various practical scenarios such as video compression to achieve better performance.

READ FULL TEXT

page 1

page 7

page 8

page 11

page 12

page 13

page 14

11/12/2020

RIFE: Real-Time Intermediate Flow Estimation for Video Frame Interpolation

We propose RIFE, a Real-time Intermediate Flow Estimation algorithm for ...
01/25/2022

Splatting-based Synthesis for Video Frame Interpolation

Frame interpolation is an essential video processing technique that adju...
03/19/2016

Learning Image Matching by Simply Watching Video

This work presents an unsupervised learning based approach to the ubiqui...
11/25/2022

Efficient Feature Extraction for High-resolution Video Frame Interpolation

Most deep learning methods for video frame interpolation consist of thre...
12/08/2016

A Maximum A Posteriori Estimation Framework for Robust High Dynamic Range Video Synthesis

High dynamic range (HDR) image synthesis from multiple low dynamic range...
07/25/2018

Flow-Grounded Spatial-Temporal Video Prediction from Still Images

Existing video prediction methods mainly rely on observing multiple hist...
02/23/2021

Deep Deformation Detail Synthesis for Thin Shell Models

In physics-based cloth animation, rich folds and detailed wrinkles are a...