LIFI: Towards Linguistically Informed Frame Interpolation

10/30/2020
by   Aradhya Neeraj Mathur, et al.
12

In this work, we explore a new problem of frame interpolation for speech videos. Such content today forms the major form of online communication. We try to solve this problem by using several deep learning video generation algorithms to generate the missing frames. We also provide examples where computer vision models despite showing high performance on conventional non-linguistic metrics fail to accurately produce faithful interpolation of speech. With this motivation, we provide a new set of linguistically-informed metrics specifically targeted to the problem of speech videos interpolation. We also release several datasets to test computer vision video generation models of their speech understanding.

READ FULL TEXT

page 2

page 4

page 9

research
06/04/2017

Deep Frame Interpolation

This work presents a supervised learning based approach to the computer ...
research
11/02/2019

Quadratic video interpolation

Video interpolation is an important problem in computer vision, which he...
research
09/20/2018

Implementing Adaptive Separable Convolution for Video Frame Interpolation

As Deep Neural Networks are becoming more popular, much of the attention...
research
02/28/2022

Learning Cross-Video Neural Representations for High-Quality Frame Interpolation

This paper considers the problem of temporal video interpolation, where ...
research
04/25/2022

Video Frame Interpolation Based on Deformable Kernel Region

Video frame interpolation task has recently become more and more prevale...
research
07/29/2021

Video Generation from Text Employing Latent Path Construction for Temporal Modeling

Video generation is one of the most challenging tasks in Machine Learnin...
research
04/06/2020

Deep Space-Time Video Upsampling Networks

Video super-resolution (VSR) and frame interpolation (FI) are traditiona...

Please sign up or login with your details

Forgot password? Click here to reset