Where's the Learning in Representation Learning for Compositional Semantics and the Case of Thematic Fit

by   Mughilan Muthupari, et al.
Columbia University
University of Washington
Göteborgs universitet

Observing that for certain NLP tasks, such as semantic role prediction or thematic fit estimation, random embeddings perform as well as pretrained embeddings, we explore what settings allow for this and examine where most of the learning is encoded: the word embeddings, the semantic role embeddings, or “the network”. We find nuanced answers, depending on the task and its relation to the training objective. We examine these representation learning aspects in multi-task learning, where role prediction and role-filling are supervised tasks, while several thematic fit tasks are outside the models' direct supervision. We observe a non-monotonous relation between some tasks' quality score and the training data size. In order to better understand this observation, we analyze these results using easier, per-verb versions of these tasks.


page 1

page 2

page 3

page 4


An Exploration of Word Embedding Initialization in Deep-Learning Tasks

Word embeddings are the interface between the world of discrete units of...

Extrapolation in NLP

We argue that extrapolation to examples outside the training space will ...

Analysis and Prediction of NLP Models Via Task Embeddings

Task embeddings are low-dimensional representations that are trained to ...

Thematic fit bits: Annotation quality and quantity for event participant representation

Modeling thematic fit (a verb–argument compositional semantics task) cur...

Deep Automated Multi-task Learning

Multi-task learning (MTL) has recently contributed to learning better re...

MoRTy: Unsupervised Learning of Task-specialized Word Embeddings by Autoencoding

Word embeddings have undoubtedly revolutionized NLP. However, pre-traine...

Please sign up or login with your details

Forgot password? Click here to reset