Seq2Seq2Sentiment: Multimodal Sequence to Sequence Models for Sentiment Analysis

07/11/2018
by   Hai Pham, et al.
0

Multimodal machine learning is a core research area spanning the language, visual and acoustic modalities. The central challenge in multimodal learning involves learning representations that can process and relate information from multiple modalities. In this paper, we propose two methods for unsupervised learning of joint multimodal representations using sequence to sequence (Seq2Seq) methods: a Seq2Seq Modality Translation Model and a Hierarchical Seq2Seq Modality Translation Model. We also explore multiple different variations on the multimodal inputs and outputs of these seq2seq models. Our experiments on multimodal sentiment analysis using the CMU-MOSI dataset indicate that our methods learn informative multimodal representations that outperform the baselines and achieve improved performance on multimodal sentiment analysis, specifically in the Bimodal case where our model is able to improve F1 Score by twelve points. We also discuss future directions for multimodal Seq2Seq methods.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
12/19/2018

Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities

Multimodal sentiment analysis is a core research area that studies speak...
research
09/07/2020

TransModality: An End2End Fusion Method with Transformer for Multimodal Sentiment Analysis

Multimodal sentiment analysis is an important research area that predict...
research
12/21/2019

Look, Read and Feel: Benchmarking Ads Understanding with Multimodal Multitask Learning

Given the massive market of advertising and the sharply increasing onlin...
research
06/16/2022

Multi-scale Cooperative Multimodal Transformers for Multimodal Sentiment Analysis in Videos

Multimodal sentiment analysis in videos is a key task in many real-world...
research
08/21/2022

CMSBERT-CLR: Context-driven Modality Shifting BERT with Contrastive Learning for linguistic, visual, acoustic Representations

Multimodal sentiment analysis has become an increasingly popular researc...
research
10/06/2021

Unsupervised Multimodal Language Representations using Convolutional Autoencoders

Multimodal Language Analysis is a demanding area of research, since it i...
research
11/11/2021

Trustworthy Multimodal Regression with Mixture of Normal-inverse Gamma Distributions

Multimodal regression is a fundamental task, which integrates the inform...

Please sign up or login with your details

Forgot password? Click here to reset