Grounding Natural Language Instructions: Can Large Language Models Capture Spatial Information?

09/17/2021
by   Julia Rozanova, et al.
8

Models designed for intelligent process automation are required to be capable of grounding user interface elements. This task of interface element grounding is centred on linking instructions in natural language to their target referents. Even though BERT and similar pre-trained language models have excelled in several NLP tasks, their use has not been widely explored for the UI grounding domain. This work concentrates on testing and probing the grounding abilities of three different transformer-based models: BERT, RoBERTa and LayoutLM. Our primary focus is on these models' spatial reasoning skills, given their importance in this domain. We observe that LayoutLM has a promising advantage for applications in this domain, even though it was created for a different original purpose (representing scanned documents): the learned spatial features appear to be transferable to the UI grounding setting, especially as they demonstrate the ability to discriminate between target directions in natural language instructions.

READ FULL TEXT

page 2

page 5

research
03/30/2021

Grounding Open-Domain Instructions to Automate Web Support Tasks

Grounding natural language instructions on the web to perform previously...
research
08/17/2021

A Game Interface to Study Semantic Grounding in Text-Based Models

Can language models learn grounded representations from text distributio...
research
08/28/2018

Mapping Natural Language Commands to Web Elements

The web provides a rich, open-domain environment with textual, structura...
research
02/06/2023

Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning

Recent works successfully leveraged Large Language Models' (LLM) abiliti...
research
06/04/2021

Grounding 'Grounding' in NLP

The NLP community has seen substantial recent interest in grounding to f...
research
10/07/2022

See, Plan, Predict: Language-guided Cognitive Planning with Video Prediction

Cognitive planning is the structural decomposition of complex tasks into...
research
12/16/2021

Call for Customized Conversation: Customized Conversation Grounding Persona and Knowledge

Humans usually have conversations by making use of prior knowledge about...

Please sign up or login with your details

Forgot password? Click here to reset