Talk the Walk: Navigating New York City through Grounded Dialogue

07/09/2018
by   Harm de Vries, et al.
14

We introduce "Talk The Walk", the first large-scale dialogue dataset grounded in action and perception. The task involves two agents (a "guide" and a "tourist") that communicate via natural language in order to achieve a common goal: having the tourist navigate to a given target location. The task and dataset, which are described in detail, are challenging and their full solution is an open problem that we pose to the community. We (i) focus on the task of tourist localization and develop the novel Masked Attention for Spatial Convolutions (MASC) mechanism that allows for grounding tourist utterances into the guide's map, (ii) show it yields significant improvements for both emergent and natural language communication, and (iii) using this method, we establish non-trivial baselines on the full task.

READ FULL TEXT

page 2

page 14

page 17

page 21

research
11/10/2016

Learning to Play Guess Who? and Inventing a Grounded Language as a Consequence

Acquiring your first language is an incredible feat and not easily dupli...
research
04/18/2021

A recipe for annotating grounded clarifications

In order to interpret the communicative intents of an utterance, it need...
research
09/10/2021

Reference-Centric Models for Grounded Collaborative Dialogue

We present a grounded neural dialogue model that successfully collaborat...
research
07/08/2019

A Natural Language Corpus of Common Grounding under Continuous and Partially-Observable Context

Common grounding is the process of creating, repairing and updating mutu...
research
05/09/2023

Dialogue Planning via Brownian Bridge Stochastic Process for Goal-directed Proactive Dialogue

Goal-directed dialogue systems aim to proactively reach a pre-determined...
research
09/26/2021

MultiDoc2Dial: Modeling Dialogues Grounded in Multiple Documents

We propose MultiDoc2Dial, a new task and dataset on modeling goal-orient...
research
12/20/2022

Ontologically Faithful Generation of Non-Player Character Dialogues

We introduce a language generation task grounded in a popular video game...

Please sign up or login with your details

Forgot password? Click here to reset