DeepAI
Log In Sign Up

Image-Grounded Conversations: Multimodal Context for Natural Question and Response Generation

01/28/2017
by   Nasrin Mostafazadeh, et al.
0

The popularity of image sharing on social media and the engagement it creates between users reflects the important role that visual context plays in everyday conversations. We present a novel task, Image-Grounded Conversations (IGC), in which natural-sounding conversations are generated about a shared image. To benchmark progress, we introduce a new multiple-reference dataset of crowd-sourced, event-centric conversations on images. IGC falls on the continuum between chit-chat and goal-directed conversation models, where visual grounding constrains the topic of conversation to event-driven utterances. Experiments with models trained on social media data show that the combination of visual and textual context enhances the quality of generated conversational turns. In human evaluation, the gap between human performance and that of both neural and retrieval architectures suggests that multi-modal IGC presents an interesting challenge for dialogue research.

READ FULL TEXT

page 4

page 5

08/16/2021

MMChat: Multi-Modal Chat Dataset on Social Media

Incorporating multi-modal contexts in conversation is an important step ...
05/19/2022

Let's Talk! Striking Up Conversations via Conversational Visual Question Generation

An engaging and provocative question can open up a great conversation. I...
04/26/2020

Endowing Empathetic Conversational Models with Personas

Empathetic conversational models have been shown to improve user satisfa...
11/24/2021

Revisiting Contextual Toxicity Detection in Conversations

Understanding toxicity in user conversations is undoubtedly an important...
10/16/2022

NormSAGE: Multi-Lingual Multi-Cultural Norm Discovery from Conversations On-the-Fly

Norm discovery is important for understanding and reasoning about the ac...
04/13/2017

Fashion Conversation Data on Instagram

The fashion industry is establishing its presence on a number of visual-...
06/02/2020

Situated and Interactive Multimodal Conversations

Next generation virtual assistants are envisioned to handle multimodal i...