Effect of Post-processing on Contextualized Word Representations

04/15/2021
by   Hassan Sajjad, et al.
0

Post-processing of static embedding has beenshown to improve their performance on both lexical and sequence-level tasks. However, post-processing for contextualized embeddings is an under-studied problem. In this work, we question the usefulness of post-processing for contextualized embeddings obtained from different layers of pre-trained language models. More specifically, we standardize individual neuron activations using z-score, min-max normalization, and by removing top principle components using the all-but-the-top method. Additionally, we apply unit length normalization to word representations. On a diverse set of pre-trained models, we show that post-processing unwraps vital information present in the representations for both lexical tasks (such as word similarity and analogy)and sequence classification tasks. Our findings raise interesting points in relation to theresearch studies that use contextualized representations, and suggest z-score normalization as an essential step to consider when using them in an application.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
01/22/2021

Effects of Pre- and Post-Processing on type-based Embeddings in Lexical Semantic Change Detection

Lexical semantic change detection is a new and innovative research field...
research
09/28/2021

When in Doubt: Improving Classification Performance with Alternating Normalization

We introduce Classification with Alternating Normalization (CAN), a non-...
research
08/20/2018

Post-Processing of Word Representations via Variance Normalization and Dynamic Embedding

Although embedded vector representations of words offer impressive perfo...
research
11/20/2022

Conceptor-Aided Debiasing of Contextualized Embeddings

Pre-trained language models reflect the inherent social biases of their ...
research
09/09/2021

Mining Points of Interest via Address Embeddings: An Unsupervised Approach

Digital maps are commonly used across the globe for exploring places tha...
research
04/30/2021

Word-Level Alignment of Paper Documents with their Electronic Full-Text Counterparts

We describe a simple procedure for the automatic creation of word-level ...
research
10/07/2021

Learning post-processing for QRS detection using Recurrent Neural Network

Deep-learning based QRS-detection algorithms often require essential pos...

Please sign up or login with your details

Forgot password? Click here to reset