MoRTy: Unsupervised Learning of Task-specialized Word Embeddings by Autoencoding

01/10/2020
by   Nils Rethmeier, et al.
0

Word embeddings have undoubtedly revolutionized NLP. However, pre-trained embeddings do not always work for a specific task (or set of tasks), particularly in limited resource setups. We introduce a simple yet effective, self-supervised post-processing method that constructs task-specialized word representations by picking from a menu of reconstructing transformations to yield improved end-task performance (MORTY). The method is complementary to recent state-of-the-art approaches to inductive transfer via fine-tuning, and forgoes costly model architectures and annotation. We evaluate MORTY on a broad range of setups, including different word embedding methods, corpus sizes and end-task semantics. Finally, we provide a surprisingly simple recipe to obtain specialized embeddings that better fit end-tasks.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 4

05/05/2021

Evaluation Of Word Embeddings From Large-Scale French Web Content

Distributed word representations are popularly used in many tasks in nat...
04/01/2021

Evaluating Neural Word Embeddings for Sanskrit

Recently, the supervised learning paradigm's surprisingly remarkable per...
02/07/2017

How to evaluate word embeddings? On importance of data efficiency and simple supervised tasks

Maybe the single most important goal of representation learning is makin...
05/03/2020

Double-Hard Debias: Tailoring Word Embeddings for Gender Bias Mitigation

Word embeddings derived from human-generated corpora inherit strong gend...
08/14/2017

Improved Answer Selection with Pre-Trained Word Embeddings

This paper evaluates existing and newly proposed answer selection method...
03/07/2019

Creation and Evaluation of Datasets for Distributional Semantics Tasks in the Digital Humanities Domain

Word embeddings are already well studied in the general domain, usually ...
11/06/2019

Invariance and identifiability issues for word embeddings

Word embeddings are commonly obtained as optimizers of a criterion funct...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.