Video captioning is the task of describing the content in a video in natural language. With the explosion of sensors and the internet as a data carrier, automatic video understanding and captioning have become essential. It can be applied in many applications such as video surveillance, assisting visually challenged people, video retrieval, and many more. Despite having many applications, jointly modeling the spatial appearance and temporal dynamics makes it a difficult task.
Motivated by machine translation 
and image captioning[5, 6], recently, the encoder-decoder architecture has been adapted for the video captioning task [7, 8, 2, 9, 1]
. With the limited amount of Spatio-temporal data, pre-trained convents have been successful at the encoder side. The decoder exploits a variant of recurrent neural networks (LSTM, GRU) to generate more relevant caption. Some recent approaches force the decoder to capture better semantics about the video by introducing reconstruction loss, while others have used multi-modal  features and memory-based modeling .
However, a potential drawback with these methods is the training signal does not align with the standard evaluation metrics such as BLEU , METEOR , ROUGE-L , CIDER . As a result, even low training and validation loss can lead to poor metric scores and vice versa, as shown in Fig.1(b). Direct optimization over metric function is not possible due to the non-differentiable nature of the network. To this end, we propose a dynamic loss network (DLN), a transformer-based model that approximates metric function and pre-trained on external data using a self-supervised setup. Although the proposed DLN can be utilized to approximate any metric function, in our case, we approximate the BLEU score. Once trained, the DLN can be used with the video captioning model in an end-to-end manner, as shown in Fig.1(a).
Finally, we demonstrate that the feedback signals from our proposed model align with the evaluation metric, as shown in Fig.1(c).
2 Related Work
2.0.1 Video Captioning.
The main breakthrough in video captioning happened with the inception of encoder-decoder based seq2seq models. The encoder-decoder framework for video captioning was first introduced by MP-LSTM , which uses mean pooling over frame features and then decodes caption by LSTM. The temporal nature of video was first modeled by S2VT  and SA-LSTM . The former shares a single LSTM for both the encoder and the decoder, while the later uses attention over frame features along with 3D HOG features. RecNet  uses backward flow and reconstruction loss to capture better semantics where MARN  uses memory to capture correspondence between a word and its various similar visual context. More recently, ORG-TRL  has modeled object interaction via Graph convolutional network (GCN). As these methods suffer from improper training signal, some effort has already been made to mitigate this issue.
Training on evaluation metric function. Ranzato et al.  use the REINFORCE algorithm  to train an image captioning model directly on BLEU score, where Rennie et al.  use the Actor-critic method . Both methods use the reward signal, but these methods are not applicable for video captioning due to the sparse nature of the reward. Zhukov et al.  propose a differentiable lower bound of expected BLEU score where Casas et al.  reported poor training signal corresponding to their formulation of differentiable BLEU score . Unlike previous works, we leverage successful Transformer based pre-trained models to approximate the evaluation metrics.
Our proposed method follows a two-stage training process. At the first stage, the DLN is trained in a self-supervised setup, whereas at the second stage, the trained DLN is used along with the existing video captioning model. The entire process flow is in the Fig.2. During the second stage, the loss from the DLN back propagates through the encoder-decoder model and forces it to capture better representation. Moreover, the proposed loss network can combined with different encoder-decoder architectures for video captioning. In this paper, we have employed the DLN on top of MP-LSTM , SA-LSTM , S2VT , RecNet , and MARN .
The video captioning task aims at generating sentence that describes the content of a given video , where is a word in caption and is a frame in video V. Standard encoder-decoder architecture models conditional word distribution given previous tokens and video feature.
where is the learnable parameters and is the word in the sentence of length T.
To generate a proper caption, semantically rich visual features need to be extracted. An encoder is responsible for the video feature extraction, which will be fed to the downstream decoder. A typical encoding method is to extract video frame features using pre-trained 2D Convnets (Inception-v4, ResNet , etc.) and fuse them to get a fixed-length video feature. In our implementation, we follow the same encoding strategy as MP-LSTM , SA-LSTM , RecNet , and MARN .
Decoder. The decoder generates the caption word by word based on the visual features obtained from the encoder. A recurrent neural network is utilized as the backbone of the decoder because of its superior temporal modeling capability. In the proposed system, the decoder is designed using LSTM  with attention over frame features .
The encoder-decoder model is jointly trained by maximizing the log-likelihood of the sentence given the video feature.
3.2 Dynamic Loss Network (DLN)
As shown in Fig.1(a), the proposed DLN is built on top of the encoder-decoder. The DLN provides an additional training signal aligned with the evaluation metric. The proposed DLN approximates the BLEU 28] is used as the backbone for the proposed DLN. In the following, we have included a brief description on the training of DLN and also, how video captioning is achieved with the trained DLN.
Training of Dynamic Loss Network. The training of the DLN is achieved in a self-supervised manner. As external data, the caption corpus of MSCOCO  is utilized for the training of DLN, specifically, for the modeling of to
score. The training data is generated following the two strategies: (i) we perturb each sentence randomly with a p% probability to generate (candidate, reference) pair. For the perturbation, deletion and swapping are done over the word(s). (ii) we use the predicted and ground truth caption as (candidate, reference) pair during training of an image captioning model on MSCOCO data. In both the cases, ground truth ( to ) is generated using the NLTK  library. It is to be noted that, the
score is computed based on the modified N-gram precision between candidate and reference sentence.
where is the modified precision between n-grams, is a scalar weight corresponding to the and, is the brevity penalty.
The self-attention layer in the transformer network (to be more specific, transformer network with word as input) calculates the attention score between words. This characteristic makes the transformer network  a natural choice to model the BLEU score function. Although BERT  and GPT  are the state-of-the-art pre-trained transformer architecture, they are not suitable to model BLEU score due to subword input tokenization. Instead, BLEU score is modeled using TransformerXL  architecture, which works with standard word input (similar to the LSTM decoder). A regression head has been added on top of the standard TransformerXL network and trained by maximizing the log-likelihood of BLEU scores given reference and candidate sentence.
where is the learnable parameters and R,C are reference and candidate sentence respectively.
Video Captioning with Dynamic Loss Network. Once trained, the DLN is combined with the standard encoder-decoder network at the second stage of training. The proposed DLN is applied only at the training stage, so there is no run-time overhead during inference. As shown in Fig.2(b) the DLN take inputs from the output of the decoder and ground truth caption. During the backward pass the output value of DLN is added to cross entropy loss and the model is trained on the combined loss function.
At the early phase of training, cross entropy acts as a better training signal so we rely more on cross entropy loss. On the other hand, we rely more on loss from the proposed loss network at the later phase of training. To this end, the hyper-parameter is introduced. and it’s value increases along with the iterations.
4 Experiments and Results
We have conducted experiments to evaluate the performance of the proposed DLN-based video captioning on two benchmark datasets: Microsoft Research-Video to Text (MSRVTT)  and Microsoft Research Video Description Corpus (MSVD) . We have investigated the effect of DLN on the performance of the state-of-the-art methods for video captioning.
MSRVTT. MSRVTT is the largest open domain video captioning dataset with 10k videos and 20 categories. Each video clip is annotated with 20 sentences, resulting in 200k video-sentence pairs. We have followed the public benchmark splits, i.e., 6513 for training, 497 for validation, and 2990 for testing.
4.2 Implementation Details
We have uniformly sampled 28 frames per video and extracted 1536 dimensional appearance features from InceptionV4 
, pre-trained on ImageNet. At the decoder end, the hidden layer contains 512 units, and the size of the word embedding is considered 468. In the Attention-based Recurrent Decoder, the dimension of the attention module is set to 256. All the sentences longer than 30 words are truncated. Apart from these, we have followed the standard settings mentioned in the models [7, 1, 8, 9, 2] while adding DLN on top of them. For the DLN, we use 16 multi-head and 18 layers TransformerXL 
pre-trained on WikiText-103. A regression head composed of three fully connected (FC) layers are added on the top of the TransformerXL. to is modeled through the training of DLN. During both stages of training, the learning rate for DLN and the end-to-end video captioning model is set to 1e-4. Adam  is employed for optimization. The model selection is made using the validation set performance. At the test time, beam search with a beam length of 5 is used for the final caption generation.
4.3 Experimental Results
We have compared our proposed model with the existing video captioning models on MSVD and MSRVTT datasets, as shown in Table.1 and Table.2. All four popular evaluation metrics, including BLEU, METEOR, ROUGE, and CIDER are reported.
|Model||without DLN||with DLN (Ours)|
|RecNet (global) ||51.1||34.0||69.4||79.7||50||34.1||71.1||78.5|
|RecNet (local) ||52.3||34.1||69.8||80.3||51.8||34.4||69.9||81|
It is apparent that adding DLN provided significant gain to the captioning performance in all metrics. In our experiment, the training is performed with or without DLN under the same settings as described by the existing methods.
|Model||without DLN||with DLN (Ours)|
|RecNet (global) ||38.3||26.2||59.1||41.7||38.7||26.5||58.7||41.9|
|RecNet (local) ||39.1||26.6||59.3||42.7||38.9||26.7||58.9||42.7|
Significance of BLEU as a prediction function for DLN. Among the different evaluation metrics, BLEU is the most uncorrelated metric with the training signal, as apparent from Fig.1(b). Moreover, significant similarity is noticed in the mechanism of the self-attention in the transformer network and the BLEU score in Eq.3. The efficacy of the DLN model has been demonstrated in Table.1 and Table.2.
Study on the training of the DLN. The training of the DLN is performed to predict to . The novel idea of the DLN is proposed in this paper, so no benchmark results are available for this task. Hence, the qualitative analysis is performed by comparing histograms of the ground truth and the predicted values on the test set, as shown in Fig.3.
Study on the Trade-off Parameter . It is indeed to state that the trade-off parameter plays a crucial role in controlling the combined loss (Eq.5) during training. Experimentally it is observed that the augmentation of DLN () has improved the performance of the majority of the state-of-the-art video captioning models. It is also observed that the quality of the captions deteriorates with the higher values of . We empirically set initial as 0.1 and increases up to 0.5 along with iterations.
This work addresses the training signal, evaluation metric alignment mismatch problem of existing video captioning models and proposed a dynamic loss network, which models the evaluation metric under consideration (BLEU is considered in our case). The training is performed in two stages, and the experimental results on the benchmark datasets support the superiority of the proposed DLN over the existing encoder-decoder based video captioning models.
Subhashini Venugopalan and Marcus Rohrbach and Jeffrey Donahue and Raymond Mooney and Trevor Darrell and Kate Saenko. Sequence to sequence-video to text. In Proceedings of the IEEE international conference on computer vision, pp. 4534–4542. 2015.
Wenjie Pei and Jiyuan Zhang and Xiangrong Wang and Lei Ke and Xiaoyong Shen and Yu-Wing Tai. Memory-attended recurrent network for video captioning. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, pp. 8347–8356, 2019.
-  Ziqi, Zhang and Yaya Shi and Chunfeng Yuan and Bing Li and Peijin Wang and Weiming Hu and Zhengjun Zha. Object Relational Graph with Teacher-Recommended Learning for Video Captioning. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2020.
-  Bahdanau, Dzmitry, Cho, Kyunghyun, and Bengio, Yoshua. Neural machine translation by jointly learning to align and translate. arXiv:1409.0473, September 2014.
-  O. Vinyals, A. Toshev, S. Bengio and D. Erhan, ”Show and tell: A neural image caption generator,” 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, 2015, pp. 3156-3164, doi: 10.1109/CVPR.2015.7298935.
-  J. Lu, C. Xiong, D. Parikh and R. Socher, ”Knowing When to Look: Adaptive Attention via a Visual Sentinel for Image Captioning,” 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 2017, pp. 3242-3250, doi: 10.1109/CVPR.2017.345.
-  S. Venugopalan, H. Xu, J. Donahue, M. Rohrbach, R. Mooney, and K. Saenko. Translating videos to natural language using deep recurrent neural networks. In NAACLHLT, 2015.
-  Yao, Li and Torabi, Atousa and Cho, Kyunghyun and Ballas, Nicolas and Pal, Christopher and Larochelle, Hugo and Courville, Aaron. (2015). Describing Videos by Exploiting Temporal Structure. 10.1109/ICCV.2015.512.
-  Bairui Wang, Lin Ma, Wei Zhang, and Wei Liu. Reconstruction network for video captioning. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 7622–7631, 2018.
-  Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level training with recurrent neural networks. ICLR, 2015
-  Steven J. Rennie and Etienne Marcheret and Youssef Mroueh and Jerret Ross and Vaibhava Goel. Self-critical Sequence Training for Image Captioning. CoRR. abs/1612.00563, 2016.
-  Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. MIT Press, 1998.
-  Vlad Zhukov and Maksim Kretov. Differentiable lower bound for expected BLEU score. CoRR. abs/1712.04708, 2017.
-  Noe Casas and José A.R. Fonollosa and Marta R. Costa-jussà. A differentiable BLEU loss. Analysis and first results. ICLR, 2018.
-  J. Xu, T. Mei, T. Yao, and Y. Rui. Msr-vtt: A large video description dataset for bridging video and language. CVPR, June 2016
-  D. L. Chen and W. B. Dolan. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pages 190–200. Association for Computational Linguistics, 2011.
Tsung-Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollar, and C. Lawrence Zitnick. Microsoft ´ COCO: common objects in context. ECCV, 2014.
-  K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311–318. Association for Computational Linguistics, 2002.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Fei-Fei Li. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211–252, 2015.
-  Bird S, Klein E, Loper E. Natural language processing with Python. Analyzing text with the natural language toolkit.Reilly Media, Inc.” 2009.
-  Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina. Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1.Association for Computational Linguistics.
Radford, Alec and Wu, Jeff and Child, Rewon and Luan, David and Amodei, Dario and Sutskever, Ilya. Language Models are Unsupervised Multitask Learners. 2019.
-  Dai, Zihang and Yang, Zhilin and Yang, Yiming and Carbonell, Jaime and Le, Quoc and Salakhutdinov, Ruslan. (2019). Transformer-XL: Attentive Language Models beyond a Fixed-Length Context. 2978-2988. 10.18653/v1/P19-1285.
C. Szegedy, S. Ioffe, V. Vanhoucke, and A. A. Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning. In AAAI, 2017.
-  He, Kaiming and Zhang, Xiangyu and Ren, Shaoqing and Sun, Jian. (2016). Deep Residual Learning for Image Recognition. 770-778. 10.1109/CVPR.2016.90.
-  D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
-  Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan and Kaiser, Lukasz and Polosukhin, Illia. (2017). Attention Is All You Need.
-  V. Ramanishka, A. Das, D. H. Park, S. Venugopalan, L. A. Hendricks, M. Rohrbach, and K. . Multimodal video description. In ACM Multimedia Conference, 2016.
-  S. Banerjee and A. Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In ACL workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, volume 29, pages 65–72, 2005.
C.-Y. Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out: Proceedings of the ACL-04 workshop, volume 8. Barcelona, Spain, 2004.
-  R. Vedantam, C. Lawrence Zitnick, and D. Parikh. Cider: Consensus-based image description evaluation. In CVPR, 2015.