Best Vision Technologies Submission to ActivityNet Challenge 2018-Task: Dense-Captioning Events in Videos

06/25/2018 ∙ by Yuan Liu, et al. ∙ 0 ∙ share

This note describes the details of our solution to the dense-captioning events in videos task of ActivityNet Challenge 2018. Specifically, we solve this problem with a two-stage way, i.e., first temporal event proposal and then sentence generation. For temporal event proposal, we directly leverage the three-stage workflow in [13, 16]. For sentence generation, we capitalize on LSTM-based captioning framework with temporal attention mechanism (dubbed as LSTM-T). Moreover, the input visual sequence to the LSTM-based video captioning model is comprised of RGB and optical flow images. At inference, we adopt a late fusion scheme to fuse the two LSTM-based captioning models for sentence generation.

READ FULL TEXT VIEW PDF

1 Sentence Generation Model

Inspired from the recent successes of LSTM based sequence models leveraged in image/video captioning [1, 3, 5, 6, 7, 10, 11, 12, 14, 15]

, we formulate our sentence generation model in an end-to-end fashion based on LSTM which encodes the input frame/optical flow sequence into a fixed dimensional vector via temporal attention mechanism and then decodes it to each target output word. An overview of our sentence generation model is illustrated in Figure

Figure 1: The sentence generation model in our system for dense-captioning events in videos task.

Model	B@1	B@2	B@3	B@4	M	R	C
LSTM-T $_{f r a m e}$	12.71	7.24	4.01	1.99	8.99	14.67	13.82
LSTM-T $_{o p t}$	12.46	7.08	3.96	1.97	8.72	14.55	13.60
LSTM-T	13.19	7.75	4.48	2.31	9.26	15.18	14.97

Table 1: Performance on ActivityNet captions validation set, where B@

N

, M, R and C are short for BLEU@

N

, METEOR, ROUGE-L and CIDEr-D scores. All values are reported as percentage (%).

In particular, given the input video with frame and optical flow sequences, each input frame/optical flow sequence ( ${v_{i}^{(1)}}_{i = 1}^{K}$ ) is fed into a two-layer LSTM with attention mechanism. At each time step $t$ , the attention LSTM decoder firstly collects the maximum contextual information by concatenating the input word $w_{t}$ with the previous output of the second-layer LSTM unit $h_{t - 1}^{2}$ and the mean-pooled video-level representation , which will be set as the input of the first-layer LSTM unit. Hence the updating procedure for the first-layer LSTM unit is as

h_{t}^{1} = f_{1} ([h_{t - 1}^{2}, W_{s} w_{t}, ¯ ¯ ¯ v]),

(1)

where $W_{s} \in R^{D_{s}^{1} \times D_{s}}$ is the transformation matric for input word $w_{t}$ , $h_{t}^{1} \in R^{D_{h}}$ is the output of the first-layer LSTM unit, and $f_{1}$ is the updating function within the first-layer LSTM unit. Next, depending on the output $h_{t}^{1}$ of the first-layer LSTM unit, a normalized attention distribution over all the frame/optical flow features is generated as:

\begin{matrix} a_{t, i} = W_{a} [tanh (W_{f} v_{i}^{(1)} + W_{h} h_{t}^{1})], λ_{t} = s o f t m a x (a_{t}), \end{matrix}

(2)

where $a_{t, i}$ is the $i$ -th element of $a_{t}$ , $W_{a} \in R^{1 \times D_{a}}$ , $W_{f} \in R^{D_{a} \times D_{v}}$ and $W_{h} \in R^{D_{a} \times D_{h}}$ are transformation matrices. $λ_{t} \in R^{K}$ denotes the normalized attention distribution and its $i$ -th element $λ_{t, i}$

is the attention probability of

v_{i}^{(1)}

. Based on the attention distribution, we calculate the attended video-level representation

{^v}_{t} = K \sum i = 1 λ_{t, i} v_{i}^{(1)}

by aggregating all the frame/optical flow features weighted with attention. We further concatenate the attended video-level feature

{^v}_{t}

with

h_{t}^{1}

and feed them into the second-layer LSTM unit, whose updating procedure is thus given as:

h_{t}^{2} = f_{2} ([{^v}_{t}, h_{t}^{1}]),

(3)

where $f_{2}$ is the updating function within the second-layer LSTM unit. The output of the second-layer LSTM unit $h_{t}^{2}$ is leveraged to predict the next word $w_{t + 1}$

through a softmax layer. Note that the policy gradient optimization method with reinforcement learning

[4, 9] is additionally leveraged to boost the sentence generation performances specific to METEOR metric.

2 Experiments

2.1 Features and Parameter Settings

Each word in the sentence is represented as “one-hot” vector (binary index vector in a vocabulary). For the input video representations, we take the output of 2048-way $p o o l 5$ layer from P3D ResNet [8] pre-trained on Kinetics dataset [2] as frame/optical flow representation. The dimension of the hidden layer in each LSTM $D_{h}$ is set as 1,000. The dimension of the hidden layer for measuring attention distribution $D_{a}$ is set as 512.

2.2 Results

Two slightly different settings of our LSTM-T are named as LSTM-T $_{f r a m e}$ and LSTM-T $_{o p t}$ which are trained with only frame and optical flow sequence, respectively. Table 1 shows the performances of our models on ActivityNet captions validation set. The results clearly indicate that by utilizing both frame and optical flow sequences in a late fusion manner, our LSTM-T boosts up the performances.

3 Conclusions

In this challenge, we mainly focus on the dense-captioning events in videos task and present a system by leveraging the three-stage workflow for temporal event proposal and LSTM-based captioning model with temporal attention mechanism for sentence generation. One possible future research direction would be how to end-to-end formulate the whole dense-captioning events in videos system.

References

[1] J. Donahue, L. Anne Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell. Long-term recurrent convolutional networks for visual recognition and description. In CVPR, 2015.
[2] W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan, F. Viola, T. Green, T. Back, P. Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950, 2017.
[3] Y. Li, T. Yao, Y. Pan, H. Chao, and T. Mei.
Jointly localizing and describing events for dense video captioning.
In CVPR, 2018.
[4] S. Liu, Z. Zhu, N. Ye, S. Guadarrama, and K. Murphy. Optimization of image description metrics using policy gradient methods. In ICCV, 2017.
[5] Y. Pan, T. Mei, T. Yao, H. Li, and Y. Rui. Jointly modeling embedding and translation to bridge video and language. In CVPR, 2016.
[6] Y. Pan, Z. Qiu, T. Yao, H. Li, and T. Mei. Seeing bot. In SIGIR, 2017.
[7] Y. Pan, T. Yao, H. Li, and T. Mei. Video captioning with transferred semantic attributes. In CVPR, 2017.
[8] Z. Qiu, T. Yao, and T. Mei. Learning spatio-temporal representation with pseudo-3d residual networks. In ICCV, 2017.
[9] S. J. Rennie, E. Marcheret, Y. Mroueh, J. Ross, and V. Goel. Self-critical sequence training for image captioning. arXiv preprint arXiv:1612.00563, 2016.
[10] S. Venugopalan, M. Rohrbach, J. Donahue, R. Mooney, T. Darrell, and K. Saenko. Sequence to sequence-video to text. In ICCV, 2015.
[11] K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhudinov, R. Zemel, and Y. Bengio. Show, attend and tell: Neural image caption generation with visual attention. In ICML, 2015.
[12] L. Yao, A. Torabi, K. Cho, N. Ballas, C. Pal, H. Larochelle, and A. Courville. Describing videos by exploiting temporal structure. In ICCV, 2015.
[13] T. Yao, Y. Li, Z. Qiu, F. Long, Y. Pan, D. Li, and T. Mei. Msr asia msm at activitynet challenge 2017: Trimmed action recognition, temporal action proposals and dense-captioning events in videos. In CVPR ActivityNet Challenge Workshop, 2017.
[14] T. Yao, Y. Pan, Y. Li, and T. Mei. Incorporating copying mechanism in image captioning for learning novel objects. In CVPR, 2017.
[15] T. Yao, Y. Pan, Y. Li, Z. Qiu, and T. Mei. Boosting image captioning with attributes. In ICCV, 2017.
[16] Y. Zhao, Y. Xiong, L. Wang, Z. Wu, D. Lin, and X. Tang. Temporal action detection with structured segment networks. arXiv preprint arXiv:1704.06228, 2017.

Best Vision Technologies Submission to ActivityNet Challenge 2018-Task: Dense-Captioning Events in Videos

Authors

Team RUC_AIM3 Technical Report at Activitynet 2020 Task 2: Exploring Sequential Events Detection for Dense Video Captioning

Weakly Supervised Dense Event Captioning in Videos

RUC+CMU: System Report for Dense Captioning Events in Videos

Dense-Captioning Events in Videos: SYSU Submission to ActivityNet Challenge 2020

The NTT DCASE2020 Challenge Task 6 system: Automated Audio Captioning with Keywords and Sentence Length Estimation

Learning Modality Interaction for Temporal Sentence Localization and Event Captioning in Videos

Controllable Video Captioning with POS Sequence Guidance Based on Gated Fusion Network

1 Sentence Generation Model

2 Experiments

2.1 Features and Parameter Settings

2.2 Results

3 Conclusions

References

Best Vision Technologies Submission to ActivityNet Challenge 2018-Task: Dense-Captioning Events in Videos

Authors

Team RUC_AIM3 Technical Report at Activitynet 2020 Task 2: Exploring Sequential Events Detection for Dense Video Captioning

Weakly Supervised Dense Event Captioning in Videos

RUC+CMU: System Report for Dense Captioning Events in Videos

Dense-Captioning Events in Videos: SYSU Submission to ActivityNet Challenge 2020

The NTT DCASE2020 Challenge Task 6 system: Automated Audio Captioning with Keywords and Sentence Length Estimation

Learning Modality Interaction for Temporal Sentence Localization and Event Captioning in Videos

Controllable Video Captioning with POS Sequence Guidance Based on Gated Fusion Network

This week in AI

1 Sentence Generation Model

2 Experiments

2.1 Features and Parameter Settings

2.2 Results

3 Conclusions

References