Joint Moment Retrieval and Highlight Detection Via Natural Language Queries

05/08/2023
by   Richard Luo, et al.
0

Video summarization has become an increasingly important task in the field of computer vision due to the vast amount of video content available on the internet. In this project, we propose a new method for natural language query based joint video summarization and highlight detection using multi-modal transformers. This approach will use both visual and audio cues to match a user's natural language query to retrieve the most relevant and interesting moments from a video. Our approach employs multiple recent techniques used in Vision Transformers (ViTs) to create a transformer-like encoder-decoder model. We evaluated our approach on multiple datasets such as YouTube Highlights and TVSum to demonstrate the flexibility of our proposed method.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/23/2022

UMT: Unified Multi-modal Transformers for Joint Video Moment Retrieval and Highlight Detection

Finding relevant moments and highlights in videos according to natural l...
research
07/21/2020

Multi-modal Transformer for Video Retrieval

The task of retrieving video content relevant to natural language querie...
research
07/20/2021

QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queries

Detecting customized moments and highlights from videos given natural la...
research
04/06/2022

Detecting key Soccer match events to create highlights using Computer Vision

The research and data science community has been fascinated with the dev...
research
06/05/2018

Mining for meaning: from vision to language through multiple networks consensus

Describing visual data into natural language is a very challenging task,...
research
07/21/2022

Focused Decoding Enables 3D Anatomical Detection by Transformers

Detection Transformers represent end-to-end object detection approaches ...
research
07/31/2019

Use What You Have: Video Retrieval Using Representations From Collaborative Experts

The rapid growth of video on the internet has made searching for video c...

Please sign up or login with your details

Forgot password? Click here to reset