LIFI: Towards Linguistically Informed Frame Interpolation

10/30/2020
by   Aradhya Neeraj Mathur, et al.
12

In this work, we explore a new problem of frame interpolation for speech videos. Such content today forms the major form of online communication. We try to solve this problem by using several deep learning video generation algorithms to generate the missing frames. We also provide examples where computer vision models despite showing high performance on conventional non-linguistic metrics fail to accurately produce faithful interpolation of speech. With this motivation, we provide a new set of linguistically-informed metrics specifically targeted to the problem of speech videos interpolation. We also release several datasets to test computer vision video generation models of their speech understanding.

READ FULL TEXT

page 2

page 4

page 9

06/04/2017

Deep Frame Interpolation

This work presents a supervised learning based approach to the computer ...
11/02/2019

Quadratic video interpolation

Video interpolation is an important problem in computer vision, which he...
09/20/2018

Implementing Adaptive Separable Convolution for Video Frame Interpolation

As Deep Neural Networks are becoming more popular, much of the attention...
02/28/2022

Learning Cross-Video Neural Representations for High-Quality Frame Interpolation

This paper considers the problem of temporal video interpolation, where ...
04/25/2022

Video Frame Interpolation Based on Deformable Kernel Region

Video frame interpolation task has recently become more and more prevale...
07/29/2021

Video Generation from Text Employing Latent Path Construction for Temporal Modeling

Video generation is one of the most challenging tasks in Machine Learnin...
03/19/2016

Learning Image Matching by Simply Watching Video

This work presents an unsupervised learning based approach to the ubiqui...

Code Repositories

linguistically-informed-frame-interpolation

None


view repo