Localizing Moments in Video with Temporal Language

09/05/2018
by   Lisa Anne Hendricks, et al.
2

Localizing moments in a longer video via natural language queries is a new, challenging task at the intersection of language and video understanding. Though moment localization with natural language is similar to other language and vision tasks like natural language object retrieval in images, moment localization offers an interesting opportunity to model temporal dependencies and reasoning in text. We propose a new model that explicitly reasons about different temporal segments in a video, and shows that temporal context is important for localizing phrases which include temporal language. To benchmark whether our model, and other recent video localization models, can effectively reason about temporal language, we collect the novel TEMPOral reasoning in video and language (TEMPO) dataset. Our dataset consists of two parts: a dataset with real videos and template sentences (TEMPO - Template Language) which allows for controlled studies on temporal language, and a human language dataset which consists of temporal sentences annotated by humans (TEMPO - Human Language).

READ FULL TEXT

page 1

page 3

page 9

page 11

page 12

research
08/04/2017

Localizing Moments in Video with Natural Language

We consider retrieving a specific temporal segment, or moment, from a vi...
research
06/06/2023

Prompting Large Language Models to Reformulate Queries for Moment Localization

The task of moment localization is to localize a temporal moment in an u...
research
12/08/2021

SNEAK: Synonymous Sentences-Aware Adversarial Attack on Natural Language Video Localization

Natural language video localization (NLVL) is an important task in the v...
research
04/01/2021

A Survey on Natural Language Video Localization

Natural language video localization (NLVL), which aims to locate a targe...
research
03/30/2022

AxIoU: An Axiomatically Justified Measure for Video Moment Retrieval

Evaluation measures have a crucial impact on the direction of research. ...
research
11/30/2018

MAN: Moment Alignment Network for Natural Language Moment Retrieval via Iterative Graph Adjustment

This research strives for natural language moment retrieval in long, unt...
research
06/18/2020

Video Moment Localization using Object Evidence and Reverse Captioning

We address the problem of language-based temporal localization of moment...

Please sign up or login with your details

Forgot password? Click here to reset