Learning Action Conditions from Instructional Manuals for Instruction Understanding

05/25/2022
by   Te-Lin Wu, et al.
0

The ability to infer pre- and postconditions of an action is vital for comprehending complex instructions, and is essential for applications such as autonomous instruction-guided agents and assistive AI that supports humans to perform physical tasks. In this work, we propose a task dubbed action condition inference, and collecting a high-quality, human annotated dataset of preconditions and postconditions of actions in instructional manuals. We propose a weakly supervised approach to automatically construct large-scale training instances from online instructional manuals, and curate a densely human-annotated and validated dataset to study how well the current NLP models can infer action-condition dependencies in the instruction texts. We design two types of models differ by whether contextualized and global information is leveraged, as well as various combinations of heuristics to construct the weak supervisions. Our experimental results show a >20 considering the entire instruction contexts and a >6 proposed heuristics.

READ FULL TEXT
research
05/19/2023

InstructIE: A Chinese Instruction-based Information Extraction Dataset

We introduce a new Information Extraction (IE) task dubbed Instruction-b...
research
06/24/2022

DialogID: A Dialogic Instruction Dataset for Improving Teaching Effectiveness in Online Environments

Online dialogic instructions are a set of pedagogical instructions used ...
research
02/24/2023

TUTORING: Instruction-Grounded Conversational Agent for Language Learners

In this paper, we propose Tutoring bot, a generative chatbot trained on ...
research
04/05/2023

ParroT: Translating During Chat Using Large Language Models

Large language models (LLMs) like ChatGPT and GPT-4 have exhibited remar...
research
06/27/2023

Style-transfer based Speech and Audio-visual Scene Understanding for Robot Action Sequence Acquisition from Videos

To realize human-robot collaboration, robots need to execute actions for...
research
10/16/2021

Understanding Procedural Knowledge by Sequencing Multimodal Instructional Manuals

The ability to sequence unordered events is an essential skill to compre...
research
12/09/2022

FLAG3D: A 3D Fitness Activity Dataset with Language Instruction

With the continuously thriving popularity around the world, fitness acti...

Please sign up or login with your details

Forgot password? Click here to reset