SOLVER: Scene-Object Interrelated Visual Emotion Reasoning Network

by   Jingyuan Yang, et al.

Visual Emotion Analysis (VEA) aims at finding out how people feel emotionally towards different visual stimuli, which has attracted great attention recently with the prevalence of sharing images on social networks. Since human emotion involves a highly complex and abstract cognitive process, it is difficult to infer visual emotions directly from holistic or regional features in affective images. It has been demonstrated in psychology that visual emotions are evoked by the interactions between objects as well as the interactions between objects and scenes within an image. Inspired by this, we propose a novel Scene-Object interreLated Visual Emotion Reasoning network (SOLVER) to predict emotions from images. To mine the emotional relationships between distinct objects, we first build up an Emotion Graph based on semantic concepts and visual features. Then, we conduct reasoning on the Emotion Graph using Graph Convolutional Network (GCN), yielding emotion-enhanced object features. We also design a Scene-Object Fusion Module to integrate scenes and objects, which exploits scene features to guide the fusion process of object features with the proposed scene-based attention mechanism. Extensive experiments and comparisons are conducted on eight public visual emotion datasets, and the results demonstrate that the proposed SOLVER consistently outperforms the state-of-the-art methods by a large margin. Ablation studies verify the effectiveness of our method and visualizations prove its interpretability, which also bring new insight to explore the mysteries in VEA. Notably, we further discuss SOLVER on three other potential datasets with extended experiments, where we validate the robustness of our method and notice some limitations of it.


page 1

page 2

page 4

page 9

page 11

page 13

page 14

page 16


Stimuli-Aware Visual Emotion Analysis

Visual emotion analysis (VEA) has attracted great attention recently, du...

A Circular-Structured Representation for Visual Emotion Distribution Learning

Visual Emotion Analysis (VEA) has attracted increasing attention recentl...

Seeking Subjectivity in Visual Emotion Distribution Learning

Visual Emotion Analysis (VEA), which aims to predict people's emotions t...

Building Emotional Machines: Recognizing Image Emotions through Deep Neural Networks

An image is a very effective tool for conveying emotions. Many researche...

Contextually-rich human affect perception using multimodal scene information

The process of human affect understanding involves the ability to infer ...

EmoSet: A Large-scale Visual Emotion Dataset with Rich Attributes

Visual Emotion Analysis (VEA) aims at predicting people's emotional resp...

A method for Selecting Scenes and Emotion-based Descriptions for a Robot's Diary

In this study, we examined scene selection methods and emotion-based des...

Please sign up or login with your details

Forgot password? Click here to reset