One Step at a Time: Long-Horizon Vision-and-Language Navigation with Milestones

02/14/2022
by   Chan Hee Song, et al.
0

We study the problem of developing autonomous agents that can follow human instructions to infer and perform a sequence of actions to complete the underlying task. Significant progress has been made in recent years, especially for tasks with short horizons. However, when it comes to long-horizon tasks with extended sequences of actions, an agent can easily ignore some instructions or get stuck in the middle of the long instructions and eventually fail the task. To address this challenge, we propose a model-agnostic milestone-based task tracker (M-TRACK) to guide the agent and monitor its progress. Specifically, we propose a milestone builder that tags the instructions with navigation and interaction milestones which the agent needs to complete step by step, and a milestone checker that systemically checks the agent's progress in its current milestone and determines when to proceed to the next. On the challenging ALFRED dataset, our M-TRACK leads to a notable 45 70 models.

READ FULL TEXT

page 1

page 3

page 8

research
02/18/2023

VLN-Trans: Translator for the Vision and Language Navigation Agent

Language understanding is essential for the navigation agent to follow i...
research
11/14/2021

Curriculum Learning for Vision-and-Language Navigation

Vision-and-Language Navigation (VLN) is a task where an agent navigates ...
research
10/18/2022

ULN: Towards Underspecified Vision-and-Language Navigation

Vision-and-Language Navigation (VLN) is a task to guide an embodied agen...
research
07/20/2023

Behavioral Analysis of Vision-and-Language Navigation Agents

To be successful, Vision-and-Language Navigation (VLN) agents must be ab...
research
05/10/2020

BabyWalk: Going Farther in Vision-and-Language Navigation by Taking Baby Steps

Learning to follow instructions is of fundamental importance to autonomo...
research
06/22/2023

Prompt to GPT-3: Step-by-Step Thinking Instructions for Humor Generation

Artificial intelligence has made significant progress in natural languag...
research
03/05/2019

The Regretful Agent: Heuristic-Aided Navigation through Progress Estimation

As deep learning continues to make progress for challenging perception t...

Please sign up or login with your details

Forgot password? Click here to reset