Introducing Orthogonal Constraint in Structural Probes

12/30/2020 ∙ by Tomasz Limisiewicz, et al. ∙ 5

With the recent success of pre-trained models in NLP, a significant focus was put on interpreting their representations. One of the most prominent approaches is structural probing (Hewitt and Manning, 2019), where a linear projection of language vector space is performed in order to approximate the topology of linguistic structures. In this work, we decompose this mapping into 1. isomorphic space rotation; 2. linear scaling that identifies and scales the most relevant directions. We introduce novel structural tasks to exam our method's ability to disentangle information hidden in the embeddings. We experimentally show that our approach can be performed in a multitask setting. Moreover, the orthogonal constraint identifies embedding subspaces encoding specific linguistic features and make the probe less vulnerable to memorization.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 7

page 11

page 12

Code Repositories

OrthogonalTransformerProbing

The code for ml probing research


view repo
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.