DeepAI
Log In Sign Up

On the Evaluation of Vision-and-Language Navigation Instructions

01/26/2021
by   Ming Zhao, et al.
5

Vision-and-Language Navigation wayfinding agents can be enhanced by exploiting automatically generated navigation instructions. However, existing instruction generators have not been comprehensively evaluated, and the automatic evaluation metrics used to develop them have not been validated. Using human wayfinders, we show that these generators perform on par with or only slightly better than a template-based generator and far worse than human instructors. Furthermore, we discover that BLEU, ROUGE, METEOR and CIDEr are ineffective for evaluating grounded navigation instructions. To improve instruction evaluation, we propose an instruction-trajectory compatibility model that operates without reference instructions. Our model shows the highest correlation with human wayfinding outcomes when scoring individual instructions. For ranking instruction generation systems, if reference instructions are available we recommend using SPICE.

READ FULL TEXT
09/30/2021

Language-Aligned Waypoint (LAW) Supervision for Vision-and-Language Navigation in Continuous Environments

In the Vision-and-Language Navigation (VLN) task an embodied agent navig...
07/23/2021

Adversarial Reinforced Instruction Attacker for Robust Vision-Language Navigation

Language instruction plays an essential role in the natural language gro...
11/25/2021

Less is More: Generating Grounded Navigation Instructions from Landmarks

We study the automatic generation of navigation instructions from 360-de...
07/11/2019

Effective and General Evaluation for Instruction Conditioned Navigation using Dynamic Time Warping

In instruction conditioned navigation, agents interpret natural language...
05/29/2021

Examiner: Automatically Locating Inconsistent Instructions Between Real Devices and CPU Emulators for ARM

Emulator is widely used to build dynamic analysis frameworks due to its ...
05/21/2018

A new dataset and model for learning to understand navigational instructions

In this paper, we present a state-of-the-art model and introduce a new d...
12/11/2022

Generic Tagging for RISC-V Binaries

With the widespread popularity of RISC-V – an open-source ISA – custom h...