Deep contextualized word representations

by   Matthew E. Peters, et al.
University of Washington
Allen Institute for Artificial Intelligence

We introduce a new type of deep contextualized word representation that models both (1) complex characteristics of word use (e.g., syntax and semantics), and (2) how these uses vary across linguistic contexts (i.e., to model polysemy). Our word vectors are learned functions of the internal states of a deep bidirectional language model (biLM), which is pre-trained on a large text corpus. We show that these representations can be easily added to existing models and significantly improve the state of the art across six challenging NLP problems, including question answering, textual entailment and sentiment analysis. We also present an analysis showing that exposing the deep internals of the pre-trained network is crucial, allowing downstream models to mix different types of semi-supervision signals.


Improved Word Sense Disambiguation Using Pre-Trained Contextualized Word Representations

Contextualized word representations are able to give different represent...

Learned in Translation: Contextualized Word Vectors

Computer vision has benefited from initializing multiple deep layers wit...

Learning Word Vectors for 157 Languages

Distributed word representations, or word vectors, have recently been ap...

SentiLR: Linguistic Knowledge Enhanced Language Representation for Sentiment Analysis

Most of the existing pre-trained language representation models neglect ...

Order-sensitive Shapley Values for Evaluating Conceptual Soundness of NLP Models

Previous works show that deep NLP models are not always conceptually sou...

Deep contextualized word representations for detecting sarcasm and irony

Predicting context-dependent and non-literal utterances like sarcastic a...

Question-Answering with Grammatically-Interpretable Representations

We introduce an architecture, the Tensor Product Recurrent Network (TPRN...

Code Repositories


Tensorflow implementation of contextualized word representations from bi-directional language models

view repo

Please sign up or login with your details

Forgot password? Click here to reset