How2: A Large-scale Dataset for Multimodal Language Understanding

by   Ramon Sanabria, et al.
Carnegie Mellon University

In this paper, we introduce How2, a multimodal collection of instructional videos with English subtitles and crowdsourced Portuguese translations. We also present integrated sequence-to-sequence baselines for machine translation, automatic speech recognition, spoken language translation, and multimodal summarization. By making available data and code for several multimodal natural language tasks, we hope to stimulate more research on these and similar challenges, to obtain a deeper understanding of multimodality in language processing.


UR-FUNNY: A Multimodal Language Dataset for Understanding Humor

Humor is a unique and creative communicative behavior displayed during s...

Multimodal Lecture Presentations Dataset: Understanding Multimodality in Educational Slides

Lecture slide presentations, a sequence of pages that contain text and f...

MSVD-Turkish: A Comprehensive Multimodal Dataset for Integrated Vision and Language Research in Turkish

Automatic generation of video descriptions in natural language, also cal...

Accurate Word Representations with Universal Visual Guidance

Word representation is a fundamental component in neural language unders...

ShapeWorld - A new test methodology for multimodal language understanding

We introduce a novel framework for evaluating multimodal deep learning m...

Sequence-to-Sequence Resources for Catalan

In this work, we introduce sequence-to-sequence language resources for C...

Multimodal Machine Translation through Visuals and Speech

Multimodal machine translation involves drawing information from more th...

1 Introduction

Multimodal sensory integration is an important aspect of human concept representation, language processing and reasoning [1]

. From a computational perspective, major breakthroughs in natural language processing (NLP), computer vision (CV), and automatic speech recognition (ASR) have resulted in improvements in a wide range of multimodal tasks, including visual question-answering 

[2], multimodal machine translation [3], visual dialogue [4], and grounded ASR [5]. Despite these advances, state-of-the-art computational models are nowhere near integrating multiple modalities as effectively as humans. This can be partially attributed to a lack of resources that are pervasively multimodal: existing datasets are typically focused on a single task, e.g. images and text for image captioning [6], images and text for visual-question answering [2], or speech and text for ASR [7]

. These datasets play a crucial role in the development of their fields, but their single-task nature limits the collective ability to develop general purpose artificial intelligence.

We introduce How2, a dataset of instructional videos paired with spoken utterances, English subtitles and their crowdsourced Portuguese translations, as well as English video summaries. The pervasive multimodality of How2 makes it an ideal resource for developing new models for multimodal understanding. In comparison to other multimodal resources, How2 is a naturally occurring dataset: neither the subtitles, nor the summaries have been crowdsourced. Furthermore, the visual content is inherently related to the spoken utterances. Figure 1 shows an example in which the presenter is explaining how to play a golf shot. If one only has access to the text, it is unclear whether the “green” in the subtitles refers to the colour green (“verde” in Portuguese), or the surface type (“green” in Portuguese). The textual context alone is not enough to disambiguate the meaning of the subtitles, and at the time of writing, both Google Translate and Microsoft Translator incorrectly translate “green” as “verde”. However, given additional visual context (green grass with a flag pole), or the audio context (outside with the sound of chipping a golf ball), our multimodal models can correctly interpret this utterance. See Appendix A.1 for more examples.

I’m very close to the green but I didn’t get it on the green so now I’m in this grass bunker.
Eu estou muito perto do green, mas eu não pus a bola no green, então agora estou neste bunker de grama.

In golf, get the body low in order to get underneath the golf ball when chipping out of thick grass from a side hill lie.

Figure 1: How2 contains a large variety of instructional videos with utterance-level English subtitles (in bold), aligned Portuguese translations (in italics), and video-level English summaries (in the box). Multimodality helps resolve ambiguities and improves understanding.

The value of additional modalities can also be demonstrated for ASR. Object and motion level visual cues can filter out systematic noise that co-occurs with activities. Scene information from an image can be used to learn a common auditory representational space for different environmental characteristics such as indoor vs. outdoor settings [8]. Entities in an image can also be used to adapt a language model towards a domain [9].

Together with the dataset, we also release code to perform baseline experiments on several tasks covering different subsets of How2. We find that action-level visual features improve automatic speech recognition, video summarization and speech-to-text translation. These results demonstrate the potential of the How2 dataset for future multimodal research.

2 How2 Dataset

Videos Hours Clips/Sentences Per Clip Statistics
300h train 13,168 298.2 184,949 5.8 seconds & 20 words
val 150 3.2 2,022 5.8 seconds & 20 words
test 175 3.7 2,305 5.8 seconds & 20 words
held 169 3.0 2,021 5.4 seconds & 20 words
2000h train 73,993 1,766.6 -
val 2,965 71.3 -
test 2,156 51.7 -
Table 1: Statistics of How2 dataset.

The How2 dataset consists of 79,114 instructional videos (2,000 hours in total, with an average length of  90 seconds) with English subtitles. The corpus can be (re-)created using the scripts and meta-data we made available at The website also contains information on how to obtain pre-computed features for validation or saving computation, and how to reproduce the experimental results we present using nmtpy [10].


We downloaded the videos from YouTube, along with various types of metadata, including ground-truth subtitles and summaries (referred to as “descriptions”) in English, written by the video creators. Videos were scraped from the YouTube platform using a keyword based spider as described in [11]. In order to produce a multilingual and multimodal dataset, the English subtitles were first re-segmented into full sentences, which were then aligned to the speech at the word level. The visual features were extracted from the video clips that correspond to these sentence-level alignments. The distribution of the duration of segments can be seen in Figure 1(b). See Appendix A.2 for more details on the alignment process.

To generate translations, we used the Figure Eight crowdsourcing platform. After conducting a pilot experiment with a small set of languages, we chose Portuguese as target language because of the availability of workers and the quality and reliability of the annotations performed by them. In order to reduce the amount of time it would take to annotate the dataset, we posed translation as a post-editing task: in another pilot experiment, we instructed crowd workers to “choose the best translation”

from English to Portuguese among candidate translations provided by three state-of-the-art online neural machine translation systems. We then selected the system that was preferred most often, and had crowdworkers post-edit the candidate translations. This process is still ongoing.

During annotation on Figure Eight, the worker population was restricted to those living in Portugal or Brazil. Workers were paid US$ 0.05 to watch a short video and post-edit the automatically translated Portuguese segment into correct Portuguese. Workers thus performed the annotation (and the pilots) in a multimodal setting. To ascertain worker reliability, a content word of each 5 sentences of the candidate translations was replaced by another random content word that was not part of the translation. If the word inserted was still present in the final translation, the annotations from that worker were discarded and the worker was banned from further contributing to the project.

At the time of writing, we had completed the collection of Portuguese translations for a 300h subset of the entire dataset from 200 workers (each was limited to 5,000 segments to post-edit but none of them reached this limit). We discarded and re-annotated 18% of the 300h. The total cost for data collection thus far was US$ 8,771.

In a verification experiment, we determined found that training an English-Portuguese neural MT system on 300h of machine generated training data degrades performance by about 1 BLEU point, when compared to a system that has been trained on the post-edited translations, when evaluated against expert-validated post-edited translations, showing that the approach is justified.

Topic distribution

We clustered the English subtitles using Latent Dirichlet Allocation (LDA) [12]. Upon analyzing the clusters with top words in each topic, inter-topic and intra-topic distances, we found that a good representation for the 300h subset consists of 22 topics. We hand-labeled these topics based on top words in each cluster, as shown in Figure 1(a).

(a) Topic distribution.
(b) Segment duration.
Figure 2: LDA topic distributions and segment duration for the 300h subset. The 2000h overall corpus exhibits very similar characteristics.


Table 1 presents summary statistics of the 2000h set and 300h subset: the val and test sets can be used for early-stopping, model selection and evaluation; the held set is reserved for future evaluations or challenges. The total set (i.e. 2000h) contains around 22.5M words. The tokenized training set of 300h subset contains around 3.8M (43K unique) and 3.6M (60K unique) words for English and Portuguese respectively.

3 Experiments

Baseline 19.4 54.4 36.0 53.9
Multimodal 18.0 54.4 37.2 54.9
Table 2: Results of the automatic speech recognition, machine translation, speech-to-text translation, and summarization experiments on test set. The arrows indicate direction of improvement.

To demonstrate and explore the potential of the How2 dataset, we propose several tasks and developed systems for them using a sequence-to-sequence (S2S) approach. Table 2 summarizes the baseline results on the 300h training set for all tasks; only the summarization task uses the entire 2000h set. More details can be found in the Appendix A.4.

  1. [leftmargin=1cm]

  2. Automatic speech recognition. We use an S2S model with a deep bi-directional LSTM encoder [13]. For multimodal ASR, we apply visual adaptive training [9, 5] where we re-train an ASR model by adding a linear adaptation layer which learns a video-specific bias to additively shift the speech features. All parameters of the network all jointly trained in this re-training step. The adaptation layer increases the model size by less than 1%.

  3. Machine translation. We train an S2S MT model for EnglishPortuguese using a bidirectional GRU [14]. For multimodal MT (MMT), we apply the same adaptive approach as we did for ASR but the inputs to be shifted are now word embeddings instead of speech features. The adaptation layer increases the model size by 8%.

  4. Speech-to-text translation. We directly translate from English speech to Portuguese using the same ASR architecture but with a different target vocabulary, which is similar to previous approaches [15, 16]. For multimodal STT, we apply the same adaptive scheme as in ASR.

  5. Summarization. The baseline is again the same S2S MT model. For multimodal summarization, we follow the hierarchical attention approach [17, 18] to combine textual and visual modalities by using a sequence of action-level features instead of an average-pooled one as in the other experiments. This latter increases the model size by 14%.

4 Related work

Task Dataset Languages Audio Visual Size
IC Flickr8K [19] EN, TR [20], ZH [21] ✓ (I) 8K
IC Flickr30K [22] 150K EN, DE [23] ✓ (I) 30K
IC MSCOCO [6] 414K EN ✓ (I) 82K
IC 820K JA [24] ✓ (I) 164K
MMT Multi30K [23] EN, DE, FR [25], CZ [26] ✓ (I) 30K
VD MSVD [27] 122K total in many ✓ (V) 5.3 hours
VD LSMDC [28] EN ✓ (V) 150 hours
VD MSRVTT [29] EN ✓ (V) 41 hours
AV-ASR Grid [30] EN ✓ (V) 50 hours
AV-ASR ViaVoice [31] EN ✓ (V) 34.9 hours
AV-ASR LRW [32] EN ✓ (V) 800 hours
STT Fisher [33] EN, ES 150 hours
STT Audiobooks [34] EN, FR 236 hours
SUM CNN/DMC [35, 36] EN 286,817 pairs
SUM DUC [37] EN 500 pairs
SUM [38] EN CZ ✓(V) 492,402 pairs
Table 3: Comparison with previous datasets: (IC) and (VD) stand for image and video captioning. Language names are encoded in ISO-639-1.

Lying at the intersection of NLP and CV [39], image captioning (IC) is the multimodal task with the largest number of datasets available. The most widely used datasets in this field are the ones with human crowdsourced descriptions, such as Flickr8K [19], Flickr30K [40], MSCOCO [6] and their extensions to other languages. A closely related task to IC is multimodal machine translation. So far, MMT has been addressed using captioning datasets extended with translations in different languages such as IAPR-TC12 [41] and Multi30K which is an extension of Flickr30K into German [23], French [25], and Czech [26]. One major pitfall of these datasets is that they lack syntactic and semantic diversity.

A similar task to IC is that of automatically describing videos (VD). The most popular datasets for VD are MSR-VTT [29], LSMDC [28], and Microsoft Research Video Description (MSVD) corpus [27] which is the only multilingual resource of this type providing 122K crowdsourced descriptions. However, two-thirds of the descriptions are in English and the ones in other languages are not parallel. How2 offers a larger amount of data, all of which is in two languages.

Lipreading can be seen as a form of multimodal ASR, albeit not fusing information at the semantic level. Popular and large-scale datasets include Grid [30] and Lip Reading in The Wild [32]. How2 is the first dataset that allows to perform multimodal ASR, using images as acoustic and linguistic context to improve accuracy. How2 is also a valuable resource for speech-to-text translation, which is otherwise often performed using Fisher-Callhome [33] and Audiobooks [34]. How2 is the only corpus for multimodal STT currently available.

Multimodal neural abstractive summarization is an emerging field for which there are no well established benchmarking datasets yet. Li et al. [38] collected a multimodal corpus of news articles containing 500 videos of English news articles paired with human annotated summaries. UzZaman et al. [42] collected a corpus of images, structured text and simplified compressed text for summarization of complex sentences. More traditional text-based summarization is commonly based on CNN/Daily Mail [35, 36], Gigaword [43] and the Document Understanding Conference challenge data [37]. An older version and non-released version of How2 was used for experiments on learning action examples in videos in [11].

5 Conclusions

We have introduced How2, a multimodal collection of instructional videos with English subtitles and crowdsourced Portuguese translations. We have also presented sequence-to-sequence baselines for machine translation, automatic speech recognition, spoken language translation, and multimodal summarization. By making available data and code for several multimodal natural language tasks, we hope to stimulate more research on these and similar challenges to obtain a deeper understanding of multimodality in language processing.


This work was mostly conducted at the 2018 Frederick Jelinek Memorial Summer Workshop on Speech and Language Technologies,111 hosted and sponsored by Johns Hopkins University. Lucia Specia received funding from the MultiMT (H2020 ERC Starting Grant No. 678017) and MMVC (Newton Fund Institutional Links Grant, ID 352343575) projects. Loïc Barrault and Ozan Caglayan received funding from the CHISTERA M2CR (No. ANR-15-CHR2-0006-01).


  • Barsalou et al. [2003] Lawrence W Barsalou, W Kyle Simmons, Aron K Barbey, and Christine D Wilson. Grounding conceptual knowledge in modality-specific systems. Trends in Cognitive Sciences, 2003.
  • Antol et al. [2015] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. VQA: Visual Question Answering. In Proceedings of the International Conference on Computer Vision (ICCV). IEEE, 2015.
  • Specia et al. [2016] Lucia Specia, Stella Frank, Khalil Sima’an, and Desmond Elliott. A shared task on multimodal machine translation and crosslingual image description (WMT). In Proceedings of the First Conference on Machine Translation (WMT). ACL, 2016.
  • Das et al. [2017] Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José M.F. Moura, Devi Parikh, and Dhruv Batra. Visual Dialog. In

    Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR)

    . IEEE, 2017.
  • Palaskar et al. [2018] Shruti Palaskar, Ramon Sanabria, and Florian Metze. End-to-end multimodal speech recognition. In Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018.
  • Chen et al. [2015] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO captions: Data collection and evaluation server. Computing Research Repository (CoRR), 2015.
  • Godfrey et al. [1992] John J Godfrey, Edward C Holliman, and Jane McDaniel. Switchboard: Telephone speech corpus for research and development. In Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 1992.
  • Miao and Metze [2016] Yajie Miao and Florian Metze.

    Open-domain audio-visual speech recognition: A deep learning approach.

    In Proceedings of Interspeech. ISCA, 2016.
  • Gupta et al. [2017] Abhinav Gupta, Yajie Miao, Leonardo Neves, and Florian Metze. Visual features for context-aware speech recognition. In Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2017.
  • Caglayan et al. [2017] Ozan Caglayan, Mercedes García-Martínez, Adrien Bardet, Walid Aransa, Fethi Bougares, and Loïc Barrault. NMTPY: A flexible toolkit for advanced neural machine translation systems. The Prague Bulletin of Mathematical Linguistics, 2017.
  • Yu et al. [2014] Shoou-I Yu, Lu Jiang, and Alexander Hauptmann. Instructional videos for unsupervised harvesting and learning of action examples. In Proceedings of the International Multimedia Conference (ACMM). ACM, 2014.
  • Blei et al. [2003] David M Blei, Andrew Y Ng, and Michael I Jordan. Latent dirichlet allocation.

    Journal of Machine Learning Research (JMLR)

    , 2003.
  • [13] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation.
  • Cho et al. [2014] Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder–decoder for statistical machine translation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). ACL, 2014.
  • Weiss et al. [2017] Ron J Weiss, Jan Chorowski, Navdeep Jaitly, Yonghui Wu, and Zhifeng Chen. Sequence-to-sequence models can directly translate foreign speech. In Proceedings of Interspeech. ISCA, 2017.
  • Bérard et al. [2018] Alexandre Bérard, Laurent Besacier, Ali Can Kocabiyikoglu, and Olivier Pietquin. End-to-end automatic speech translation of audiobooks. In Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018.
  • Libovický and Helcl [2017] Jindřich Libovický and Jindřich Helcl. Attention strategies for multi-source sequence-to-sequence learning. In Proceedings Annual Meeting of the Association for Computational Linguistics (ACL). ACL, 2017.
  • Libovický et al. [2018] Jindřich Libovický, Shruti Palaskar, Spandana Gella, and Florian Metze. Multimodal abstractive summarization of open-domain videos. In Proceedings of the Workshop on Visually Grounded Interaction and Language (ViGIL). NIPS, 2018.
  • Hodosh et al. [2013] Micah Hodosh, Peter Young, and Julia Hockenmaier.

    Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics.

    Journal of Artificial Intelligence Research (JAIR), 2013.
  • Unal et al. [2016] Mesut Erhan Unal, Begum Citamak, Semih Yagcioglu, Aykut Erdem, Erkut Erdem, Nazli Ikizler Cinbis, and Ruket Cakici. Tasviret: Görüntülerden otomatik türkçe açıklama oluşturma İçin bir denektaçı veri kümesi (TasvirEt: A benchmark dataset for automatic Turkish description generation from images). In Proceesdings of the Sinyal İşleme ve İletişim Uygulamaları Kurultayı (SIU 2016). IEEE, 2016.
  • Li et al. [2016] Xirong Li, Weiyu Lan, Jianfeng Dong, and Hailong Liu. Adding Chinese captions to images. In Proceedings of the International Conference on Multimedia Retrieval (ICMR). ACM, 2016.
  • Plummer et al. [2017] Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. International Journal of Computer Vision, 2017.
  • Elliott et al. [2016] Desmond Elliott, Stella Frank, Khalil Sima’an, and Lucia Specia. Multi30k: Multilingual english-german image descriptions. In Proceedings of the Workshop on Vision and Language. ACL, 2016.
  • Yoshikawa et al. [2017] Yuya Yoshikawa, Yutaro Shigeto, and Akikazu Takeuchi. STAIR captions: Constructing a large-scale Japanese image caption dataset. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL). ACL, 2017.
  • Elliott et al. [2018] Desmond Elliott, Stella Frank, Loïc Barrault, Fethi Bougares, and Lucia Specia. Findings of the second shared task on multimodal machine translation and multilingual image description. In Proceedings of the Second Conference on Machine Translation (WMT). ACL, 2018.
  • Barrault et al. [2018] Loïc Barrault, Fethi Bougares, Lucia Specia, Chiraag Lala, Desmond Elliott, and Stella. Frank. Findings of the shared task on multimodal machine translation (WMT). In Proceedings of Conference on Machine Translation (WMT). ACL, 2018.
  • Chen and Dolan [2011] David L. Chen and William B. Dolan. Building a persistent workforce on Mechanical Turk for multilingual data collection. In Proceedings of The 3rd Human Computation Workshop (HCOMP). AAAI, 2011.
  • Rohrbach et al. [2017] Anna Rohrbach, Atousa Torabi, Marcus Rohrbach, Niket Tandon, Chris Pal, Hugo Larochelle, Aaron Courville, and Bernt Schiele. Movie description. International Journal of Computer Vision, 2017.
  • Xu et al. [2016] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. MSR-VTT: a large video description dataset for bridging video and language. In Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2016.
  • Cooke et al. [2006] Martin Cooke, Jon Barker, Stuart Cunningham, and Xu Shao. An audio-visual corpus for speech perception and automatic speech recognition. The Journal of the Acoustical Society of America, 2006.
  • Neti et al. [2000] Chalapathy Neti, Gerasimos Potamianos, Juergen Luettin, Iain Matthews, Herve Glotin, Dimitra Vergyri, June Sison, and Azad Mashari. Audio visual speech recognition. Technical report, IDIAP, 2000.
  • Chung and Zisserman [2016] Joon Son Chung and Andrew Zisserman. Lip reading in the wild. In Proceedings of the Asian Conference on Computer Vision (ACCV). Springer, 2016.
  • Post et al. [2013] Matt Post, Gaurav Kumar, Adam Lopez, Damianos Karakos, Chris Callison-Burch, and Sanjeev Khudanpur. Improved Speech-to-Text translation with the Fisher and Callhome Spanish-English speech translation corpus. In Proceedings International Workshop on Spoken Language Translation (IWSLT). ACL, 2013.
  • Kocabiyikoglu et al. [2018] Ali Can Kocabiyikoglu, Laurent Besacier, and Olivier Kraif. Augmenting Librispeech with French Translations: A Multimodal Corpus for Direct Speech Translation Evaluation. In Proceedings of the International Conference on Language Resources and Evaluation (LREC). ELRA, 2018.
  • Hermann et al. [2015] Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In Proceedings of the International Conference on Neural Information Processing Systems (NIPS). NIPS, 2015.
  • Nallapati et al. [2016] Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çaglar G ulçehre, and Bing Xiang.

    Abstractive text summarization using sequence-to-sequence rnns and beyond.

    In Proceedings of the Conference on Computational Natural Language Learning (CoNLL). ACL, 2016.
  • Over et al. [2007] Paul Over, Hoa Dang, and Donna Harman. Duc in context. Information Processing & Management, 2007.
  • Li et al. [2017] Haoran Li, Junnan Zhu, Cong Ma, Jiajun Zhang, and Chengqing Zong. Multi-modal summarization for asynchronous collection of text, image, audio and video. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). ACL, 2017.
  • Bernardi et al. [2016] Raffaella Bernardi, Ruken Cakici, Desmond Elliott, Aykut Erdem, Erkut Erdem, Nazli Ikizler-Cinbis, Frank Keller, Adrian Muscat, and Barbara Plank. Automatic description generation from images: A survey of models, datasets, and evaluation measures. Journal of Artificial Intelligence Research (JAIR), 2016.
  • Young et al. [2014] Peter Young, Alice Lai, Micha Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics (TACL), 2014.
  • Grubinger et al. [2006] Michael Grubinger, Paul D. Clough, Henning Muller, and Thomas Desealers. The IAPR TC-12 benchmark: A new evaluation resource for visual information systems. In Proceedings of International Conference on Language Resources and Evaluation (LREC). ELRA, 2006.
  • UzZaman et al. [2011] Naushad UzZaman, Jeffrey P Bigham, and James F Allen. Multimodal summarization of complex sentences. In Proceedings International Conference on Intelligent User Interfaces (IUI). ACM, 2011.
  • Napoles et al. [2012] Courtney Napoles, Matthew Gormley, and Benjamin Van Durme. Annotated gigaword. In Proceedings of the Joint Workshop on Automatic Knowledge Base Construction and Web-scale Knowledge Extraction. ACL, 2012.
  • Povey et al. [2011] Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, Jan Silovsky, Georg Stemmer, and Karel Vesely. The Kaldi speech recognition toolkit. In Workshop on Automatic Speech Recognition and Understanding (ASRU). IEEE, 2011.
  • Hara et al. [2018] Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh.

    Can spatiotemporal 3d CNNs retrace the history of 2d cnns and imagenet?

    In Proceedings of the Conference on Computer Vision and Pattern Recognition(CVPR). IEEE, 2018.
  • [46] Taku Kudo. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP). ACL.
  • Chan et al. [2016] William Chan, Navdeep Jaitly, Quoc V. Le, and Oriol Vinyals.

    Listen, attend and spell: A neural network for large vocabulary conversational speech recognition.

    In Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2016.
  • Sennrich et al. [2017] Rico Sennrich, Orhan Firat, Kyunghyun Cho, Alexandra Birch-Mayne, Barry Haddow, Julian Hitschler, Marcin Junczys-Dowmunt, Samuel Läubli, Antonio Miceli Barone, Jozef Mokry, and Maria Nadejde. Nematus: a toolkit for neural machine translation. In Proceedings of the European Chapter of the Association for Computational Linguistics (EACL). Software Demonstrations. ACL, 2017.
  • Bahdanau et al. [2014] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. Computing Research Repository (CoRR), 2014.
  • Press and Wolf [2017] Ofir Press and Lior Wolf. Using the output embedding to improve language models. In Proceedings of the Conference of the European Chapter of the Association for Computational Linguistics (EACL). ACL, 2017.
  • Srivastava et al. [2014] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research (JMLR), 2014.
  • Kingma and Ba [2014] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. Journal of Machine Learning Research (JMLR), 2014.
  • Pascanu et al. [2013] Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio.

    On the difficulty of training recurrent neural networks.

    In Proceedings of the International Conference on International Conference on Machine Learning (ICML). IMIS, 2013.
  • Papineni et al. [2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: A method for automatic evaluation of machine translation. In Proceedings of the Annual Meeting on Association for Computational Linguistics (ACL). ACL, 2002.
  • Lin and Och [2004] Chin-Yew Lin and Franz Josef Och. Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. In Proceedings of the Meeting of the Association for Computational Linguistics (ACL). ACL, 2004.

Appendix A Appendix

a.1 How2 Examples

In Figure 3, we list three typical instances from the How2 dataset. In these examples, we can see the correspondence between the content of the video frame, the summary, and the utterance. This multimodal correspondence is what systems can exploit by using the How2 dataset. In the first example, a hairdresser is explaining how to use a specific hair product. In that case, the visual elements (i.e., hair product, hairdresser) and the scene, a hairdressing salon, are a rich source of context. In the second example, a woman is cooking in a kitchen with many cooking devices. In the third example, we can infer the acoustic environment (i.e., outdoors) by the scene.

Actually I use this beautiful moroccan oil and it’s really wonderful on the hair.
Na verdade eu uso este belo óleo marroquino e é realmente maravilhoso no cabelo.

Relaxed African-American hair should be moisturized and washed to maintain good and healthy hair. Care for African-American, biracial or ethnic hair with tips from a professional hairstylist in this free video on hair care.

Like said you can cook this pretty quickly with your family.
Como disse você pode cozinhar isso muito rapidamente com sua família.

Learn how to cook and serve picadillo con arroz with expert cooking tips in this free classic Cuban recipe video clip.

When your wide receivers get into their stance they want to have one foot forward, they want to have good position and be ready to fire off the line.
Quando seus receptores largos entram em sua posição eles devem ter um pé para a frent,e eles devem ter uma boa posição e estar pronto para disparar a linha.

Learn some great tips on how to line up as a receiver in this free video clip on how to play football.

Figure 3: Examples of the How2 dataset that reflect its sample variability and modality correlation. The image is a randomly selected frame from a concrete segment of the video, the text in bold is the utterance pronounced during this segment, the Portuguese text in italic are the translations corresponding to the utterance and finally, the summary of the whole video is placed inside the rectangle.

a.2 Modality Alignment and Data Checks

To combine all modalities (i.e., speech, video, transcriptions, and translations) successfully, we need to establish and validate their correspondence in time. While the audio transcription is generated from subtitles, these do not always correspond to the actual audio. We thus decided to generate token-level (e.g. word-level) time stamps that link text, audio, and video modality. From these, utterance-level start and end times were also calculated.

To align text and audio, we perform a Viterbi alignment between the transcriptions, which were provided by the users who uploaded the videos, and the audio track of How2, using Kaldi’s [44]

Wall Street Journal (WSJ) GMM/ HMM acoustic model. This alignment process estimates the start and end times of each sentence in the audio track. Finally, by using the estimated alignments, we can segment the audio and video track according to the utterances.

To make sure the data will be suitable for the proposed use, we validated two properties: First, we verified that the word alignment is indeed accurate by manually inspecting randomly chosen utterances. The 300 h sub-set was selected to give good alignment scores, and the WSJ model seemed to perform best in that respect: “good” (according to the score) utterances were indeed accurately aligned, when compared to alignments generated with other models, including those developed for speech synthesis. Second, the “transcription” data has been generated from video subtitles, which were not meant to be verbatim and highly accurate “transliterations” of the spoken content. Rather, the “transcription” text is a somewhat canonical form of the spoken word, which is fine for our proposed uses, although it may lead to slightly higher overall word error rates for the speech-to-text tasks.

a.3 Feature Extraction and Processing

Speech Features

We used Kaldi [44] to extract 40-dimensional filter bank features from 16kHz raw speech using a time window of 25ms with 10ms frame shift and concatenated 3-dimensional pitch features to obtain the final 43-dimensional speech features. A per-video

Cepstral Mean and Variance Normalization (CMVN) is further applied to account for speaker variability.

Visual Features

A 2048-dimensional feature vector per 16 frames is extracted from the videos using a CNN trained to recognize 400 different actions

[45]. It should be noted that this results in a sequence of feature vectors per video rather than a single/global one. In order to obtain the latter, we average pooled the extracted features into a single 2048-dimensional feature vector which will represent all sentences segmented out of a single video.

Text Features

All texts are normalized, lowercased and filtered from punctuation. A SentencePiece model [46] is learned separately for English and Portuguese resulting into vocabularies of 5K each except for summarization which uses word-level tokens.

a.4 Architecture Details

Automatic Speech Recognition

We use a 6-layer pyramidal encoder [47] (with interleaved projection layers) where the middle two layers skip every other input resulting into a subsampling rate of 4. The decoder is a 2-layer conditional GRU (CGRU) decoder [48], a transitive decoder where the hidden state of the second GRU is fed back to the first GRU. The first GRU is initialized with the mean encoder hidden state transformed using a layer. A feed-forward attention mechanism [49] is used inside the decoder and the input and output embeddings are tied [50].

Multimodal Automatic Speech Recognition

For multimodal ASR, we apply video adaptive training [9, 5]

with a learned linear transformation of the visual feature vector

that will act as a visual bias over a given speech feature at time . The shifted speech feature is thus computed as follows:

To train this model, we first initialize the model parameters using a previously trained ASR and jointly optimize all parameters including and above.

Machine Translation

We train a sequence-to-sequence neural MT model with a 2-layer bidirectional GRU [14] encoder and a 2-layer conditional GRU decoder[48]. A dropout [51] with

drop probability is used after source embeddings, source encodings and before the final


Multimodal Summarization

We follow the hierarchical attention approach [17] to combine textual and visual modalities. Unlike previous multimodal ASR, MT and STT architectures, the visual features described in Section 2 are now used as-is, i.e. as a sequence of 2048-dimensional vectors rather than being average pooled into a single vector.

a.5 Training Details

Unless otherwise specified, we use ADAM [52] as the optimizer with an initial learning rate of 0.0004. The gradients are clipped to have unit norm [53]

. The training is early stopped if the task performance on validation set does not improve for 10 consecutive epochs. Task performance is assessed using Word Error Rate (WER) for speech recognition, BLEU

[54] for translation tasks and ROUGE-L [55] for the summarization task. The learning rate is halved whenever the task performance does not improve for two consecutive epochs. All systems are trained three times with different random initializations. The hypotheses are decoded using beam search with a beam size of 10. We report the average results of the three runs. We use nmtpytorch [10] to train the models.