Captioning Near-Future Activity Sequences

by   Tahmida Mahmud, et al.

Most of the existing works on human activity analysis focus on recognition or early recognition of the activity labels from complete or partial observations. Similarly, existing video captioning approaches focus on the observed events in videos. Predicting the labels and the captions of future activities where no frames of the predicted activities have been observed is a challenging problem, with important applications that require anticipatory response. In this work, we propose a system that can infer the labels and the captions of a sequence of future activities. Our proposed network for label prediction of a future activity sequence is similar to a hybrid Siamese network with three branches where the first branch takes visual features from the objects present in the scene, the second branch takes observed activity features and the third branch captures the last observed activity features. The predicted labels and the observed scene context are then mapped to meaningful captions using a sequence-to-sequence learning based method. Experiments on three challenging activity analysis datasets and a video description dataset demonstrate that both our label prediction framework and captioning framework outperforms the state-of-the-arts.


page 2

page 3

page 5

page 11

page 13


Long-Term Anticipation of Activities with Cycle Consistency

With the success of deep learning methods in analyzing activities in vid...

Meaning guided video captioning

Current video captioning approaches often suffer from problems of missin...

The Use of Video Captioning for Fostering Physical Activity

Video Captioning is considered to be one of the most challenging problem...

Self-Regulated Learning for Egocentric Video Activity Anticipation

Future activity anticipation is a challenging problem in egocentric visi...

Learning Representations for Predicting Future Activities

Foreseeing the future is one of the key factors of intelligence. It invo...

Figure Captioning with Reasoning and Sequence-Level Training

Figures, such as bar charts, pie charts, and line plots, are widely used...

Machine-Generated Hierarchical Structure of Human Activities to Reveal How Machines Think

Deep-learning based computer vision models have proved themselves to be ...