DeepAI AI Chat
Log In Sign Up

SIMMC 2.0: A Task-oriented Dialog Dataset for Immersive Multimodal Conversations

by   Satwik Kottur, et al.

We present a new corpus for the Situated and Interactive Multimodal Conversations, SIMMC 2.0, aimed at building a successful multimodal assistant agent. Specifically, the dataset features 11K task-oriented dialogs (117K utterances) between a user and a virtual assistant on the shopping domain (fashion and furniture), grounded in situated and photo-realistic VR scenes. The dialogs are collected using a two-phase pipeline, which first generates simulated dialog flows via a novel multimodal dialog simulator we propose, followed by manual paraphrasing of the generated utterances. In this paper, we provide an in-depth analysis of the collected dataset, and describe in detail the four main benchmark tasks we propose for SIMMC 2.0. The preliminary analysis with a baseline model highlights the new challenges that the SIMMC 2.0 dataset brings, suggesting new directions for future research. Our dataset and code will be made publicly available.


page 1

page 2

page 11


Situated and Interactive Multimodal Conversations

Next generation virtual assistants are envisioned to handle multimodal i...

Navigating Connected Memories with a Task-oriented Dialog System

Recent years have seen an increasing trend in the volume of personal med...

Finding Dominant User Utterances And System Responses in Conversations

There are several dialog frameworks which allow manual specification of ...

Multimodal Dialogs (MMD): A large-scale dataset for studying multimodal domain-aware conversations

While multimodal conversation agents are gaining importance in several d...

Tell Your Story: Task-Oriented Dialogs for Interactive Content Creation

People capture photos and videos to relive and share memories of persona...

End-to-End Learning of Flowchart Grounded Task-Oriented Dialogs

We propose a novel problem within end-to-end learning of task-oriented d...

Migratable AI: Personalizing Dialog Conversations with migration context

The migration of conversational AI agents across different embodiments i...