Introducing Orthogonal Constraint in Structural Probes

by   Tomasz Limisiewicz, et al.

With the recent success of pre-trained models in NLP, a significant focus was put on interpreting their representations. One of the most prominent approaches is structural probing (Hewitt and Manning, 2019), where a linear projection of language vector space is performed in order to approximate the topology of linguistic structures. In this work, we decompose this mapping into 1. isomorphic space rotation; 2. linear scaling that identifies and scales the most relevant directions. We introduce novel structural tasks to exam our method's ability to disentangle information hidden in the embeddings. We experimentally show that our approach can be performed in a multitask setting. Moreover, the orthogonal constraint identifies embedding subspaces encoding specific linguistic features and make the probe less vulnerable to memorization.


page 7

page 11

page 12


Examining Cross-lingual Contextual Embeddings with Orthogonal Structural Probes

State-of-the-art contextual embeddings are obtained from large language ...

A Tale of a Probe and a Parser

Measuring what linguistic information is encoded in neural models of lan...

A Simple Geometric Method for Cross-Lingual Linguistic Transformations with Pre-trained Autoencoders

Powerful sentence encoders trained for multiple languages are on the ris...

A Closer Look at the Robustness of Vision-and-Language Pre-trained Models

Large-scale pre-trained multimodal transformers, such as ViLBERT and UNI...

Learning Universal Representations from Word to Sentence

Despite the well-developed cut-edge representation learning for language...

Probing BERT in Hyperbolic Spaces

Recently, a variety of probing tasks are proposed to discover linguistic...

Multi-View Pre-Trained Model for Code Vulnerability Identification

Vulnerability identification is crucial for cyber security in the softwa...

Code Repositories


The code for ml probing research

view repo