Navigating Connected Memories with a Task-oriented Dialog System

11/15/2022
by   Seungwhan Moon, et al.
0

Recent years have seen an increasing trend in the volume of personal media captured by users, thanks to the advent of smartphones and smart glasses, resulting in large media collections. Despite conversation being an intuitive human-computer interface, current efforts focus mostly on single-shot natural language based media retrieval to aid users query their media and re-live their memories. This severely limits the search functionality as users can neither ask follow-up queries nor obtain information without first formulating a single-turn query. In this work, we propose dialogs for connected memories as a powerful tool to empower users to search their media collection through a multi-turn, interactive conversation. Towards this, we collect a new task-oriented dialog dataset COMET, which contains 11.5k user<->assistant dialogs (totaling 103k utterances), grounded in simulated personal memory graphs. We employ a resource-efficient, two-phase data collection pipeline that uses: (1) a novel multimodal dialog simulator that generates synthetic dialog flows grounded in memory graphs, and, (2) manual paraphrasing to obtain natural language utterances. We analyze COMET, formulate four main tasks to benchmark meaningful progress, and adopt state-of-the-art language models as strong baselines, in order to highlight the multimodal challenges captured by our dataset.

READ FULL TEXT

page 6

page 12

page 13

research
04/18/2021

SIMMC 2.0: A Task-oriented Dialog Dataset for Immersive Multimodal Conversations

We present a new corpus for the Situated and Interactive Multimodal Conv...
research
11/08/2022

Tell Your Story: Task-Oriented Dialogs for Interactive Content Creation

People capture photos and videos to relive and share memories of persona...
research
03/31/2016

Data Collection for Interactive Learning through the Dialog

This paper presents a dataset collected from natural dialogs which enabl...
research
09/15/2021

End-to-End Learning of Flowchart Grounded Task-Oriented Dialogs

We propose a novel problem within end-to-end learning of task-oriented d...
research
04/27/2023

q2d: Turning Questions into Dialogs to Teach Models How to Search

One of the exciting capabilities of recent language models for dialog is...
research
05/23/2023

WikiChat: A Few-Shot LLM-Based Chatbot Grounded with Wikipedia

Despite recent advances in Large Language Models (LLMs), users still can...
research
09/14/2020

At your Command! An Empirical Study on How LaypersonsTeach Robots New Functions

Even though intelligent systems such as Siri or Google Assistant are enj...

Please sign up or login with your details

Forgot password? Click here to reset