On the Limits of Learning to Actively Learn Semantic Representations

by   Omri Koshorek, et al.

One of the goals of natural language understanding is to develop models that map sentences into meaning representations. However, training such models requires expensive annotation of complex structures, which hinders their adoption. Learning to actively-learn (LTAL) is a recent paradigm for reducing the amount of labeled data by learning a policy that selects which samples should be labeled. In this work, we examine LTAL for learning semantic representations, such as QA-SRL. We show that even an oracle policy that is allowed to pick examples that maximize performance on the test set (and constitutes an upper bound on the potential of LTAL), does not substantially improve performance compared to a random policy. We investigate factors that could explain this finding and show that a distinguishing characteristic of successful applications of LTAL is the interaction between optimization and the oracle policy selection process. In successful applications of LTAL, the examples selected by the oracle policy do not substantially depend on the optimization procedure, while in our setup the stochastic nature of optimization strongly affects the examples selected by the oracle. We conclude that the current applicability of LTAL for improving data efficiency in learning semantic meaning representations is limited.


page 1

page 2

page 3

page 4


We went to look for meaning and all we got were these lousy representations: aspects of meaning representation for computational semantics

In this paper we examine different meaning representations that are comm...

Actively Avoiding Nonsense in Generative Models

A generative model may generate utter nonsense when it is fit to maximiz...

Learning To Retrieve Prompts for In-Context Learning

In-context learning is a recent paradigm in natural language understandi...

Improve the efficiency of deep reinforcement learning through semantic exploration guided by natural language

Reinforcement learning is a powerful technique for learning from trial a...

Acquiring Word-Meaning Mappings for Natural Language Interfaces

This paper focuses on a system, WOLFIE (WOrd Learning From Interpreted E...

Training Dynamic based data filtering may not work for NLP datasets

The recent increase in dataset size has brought about significant advanc...

QActor: On-line Active Learning for Noisy Labeled Stream Data

Noisy labeled data is more a norm than a rarity for self-generated conte...

Please sign up or login with your details

Forgot password? Click here to reset