Mapping Natural Language Commands to Web Elements

08/28/2018
by   Panupong Pasupat, et al.
0

The web provides a rich, open-domain environment with textual, structural, and spatial properties. We propose a new task for grounding language in this environment: given a natural language command (e.g., "click on the second article"), choose the correct element on the web page (e.g., a hyperlink or text box). We collected a dataset of over 50,000 commands that capture various phenomena such as functional references (e.g. "find who made this site"), relational reasoning (e.g. "article by john"), and visual reasoning (e.g. "top-most article"). We also implemented and analyzed three baseline models that capture different phenomena present in the dataset.

READ FULL TEXT
research
09/17/2021

Grounding Natural Language Instructions: Can Large Language Models Capture Spatial Information?

Models designed for intelligent process automation are required to be ca...
research
09/14/2020

A Comparison of Two Fluctuation Analyses for Natural Language Clustering Phenomena: Taylor and Ebeling Neiman Methods

This article considers the fluctuation analysis methods of Taylor and Eb...
research
01/11/2019

EQUATE: A Benchmark Evaluation Framework for Quantitative Reasoning in Natural Language Inference

Quantitative reasoning is an important component of reasoning that any i...
research
01/02/2020

RDF 1.1: Knowledge Representation and Data Integration Language for the Web

Resource Description Framework (RDF) can seen as a solution in today's l...
research
11/01/2018

A Corpus for Reasoning About Natural Language Grounded in Photographs

We introduce a new dataset for joint reasoning about language and vision...
research
12/10/2017

Learning Interpretable Spatial Operations in a Rich 3D Blocks World

In this paper, we study the problem of mapping natural language instruct...
research
07/02/2023

HeGeL: A Novel Dataset for Geo-Location from Hebrew Text

The task of textual geolocation - retrieving the coordinates of a place ...

Please sign up or login with your details

Forgot password? Click here to reset