DeepAI AI Chat
Log In Sign Up

MoRTy: Unsupervised Learning of Task-specialized Word Embeddings by Autoencoding

01/10/2020
by   Nils Rethmeier, et al.
0

Word embeddings have undoubtedly revolutionized NLP. However, pre-trained embeddings do not always work for a specific task (or set of tasks), particularly in limited resource setups. We introduce a simple yet effective, self-supervised post-processing method that constructs task-specialized word representations by picking from a menu of reconstructing transformations to yield improved end-task performance (MORTY). The method is complementary to recent state-of-the-art approaches to inductive transfer via fine-tuning, and forgoes costly model architectures and annotation. We evaluate MORTY on a broad range of setups, including different word embedding methods, corpus sizes and end-task semantics. Finally, we provide a surprisingly simple recipe to obtain specialized embeddings that better fit end-tasks.

READ FULL TEXT
05/05/2021

Evaluation Of Word Embeddings From Large-Scale French Web Content

Distributed word representations are popularly used in many tasks in nat...
04/01/2021

Evaluating Neural Word Embeddings for Sanskrit

Recently, the supervised learning paradigm's surprisingly remarkable per...
10/28/2022

Analyzing Acoustic Word Embeddings from Pre-trained Self-supervised Speech Models

Given the strong results of self-supervised models on various tasks, the...
02/07/2017

How to evaluate word embeddings? On importance of data efficiency and simple supervised tasks

Maybe the single most important goal of representation learning is makin...
11/06/2019

Invariance and identifiability issues for word embeddings

Word embeddings are commonly obtained as optimizers of a criterion funct...
03/07/2019

Creation and Evaluation of Datasets for Distributional Semantics Tasks in the Digital Humanities Domain

Word embeddings are already well studied in the general domain, usually ...
08/09/2022

Where's the Learning in Representation Learning for Compositional Semantics and the Case of Thematic Fit

Observing that for certain NLP tasks, such as semantic role prediction o...