DeepAI AI Chat
Log In Sign Up

Multimodal Matching Transformer for Live Commenting

02/07/2020
by   Chaoqun Duan, et al.
Harbin Institute of Technology
0

Automatic live commenting aims to provide real-time comments on videos for viewers. It encourages users engagement on online video sites, and is also a good benchmark for video-to-text generation. Recent work on this task adopts encoder-decoder models to generate comments. However, these methods do not model the interaction between videos and comments explicitly, so they tend to generate popular comments that are often irrelevant to the videos. In this work, we aim to improve the relevance between live comments and videos by modeling the cross-modal interactions among different modalities. To this end, we propose a multimodal matching transformer to capture the relationships among comments, vision, and audio. The proposed model is based on the transformer framework and can iteratively learn the attention-aware representations for each modality. We evaluate the model on a publicly available live commenting dataset. Experiments show that the multimodal matching transformer model outperforms the state-of-the-art methods.

READ FULL TEXT

page 1

page 6

08/13/2018

Live Video Comment Generation Based on Surrounding Frames and Live Comments

In this paper, we propose the task of live comment generation. Live comm...
10/19/2022

VTC: Improving Video-Text Retrieval with User Comments

Multi-modal retrieval is an important problem for many applications, suc...
11/07/2019

Diversified Co-Attention towards Informative Live Video Commenting

We focus on the task of Automatic Live Video Commenting (ALVC), which ai...
06/04/2020

Response to LiveBot: Generating Live Video Comments Based on Visual and Textual Contexts

Live video commenting systems are an emerging feature of online video si...
01/28/2021

VX2TEXT: End-to-End Learning of Video-Based Text Generation From Multimodal Inputs

We present Vx2Text, a framework for text generation from multimodal inpu...
07/23/2020

SBAT: Video Captioning with Sparse Boundary-Aware Transformer

In this paper, we focus on the problem of applying the transformer struc...
01/17/2023

CS-lol: a Dataset of Viewer Comment with Scene in E-sports Live-streaming

Billions of live-streaming viewers share their opinions on scenes they a...