Continual Learning for Sentence Representations Using Conceptors

04/18/2019
by   Tianlin Liu, et al.
20

Distributed representations of sentences have become ubiquitous in natural language processing tasks. In this paper, we consider a continual learning scenario for sentence representations: Given a sequence of corpora, we aim to optimize the sentence encoder with respect to the new corpus while maintaining its accuracy on the old corpora. To address this problem, we propose to initialize sentence encoders with the help of corpus-independent features, and then sequentially update sentence encoders using Boolean operations of conceptor matrices to learn corpus-dependent features. We evaluate our approach on semantic textual similarity tasks and show that our proposed sentence encoder can continually learn features from new corpora while retaining its competence on previously encountered corpora.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
11/30/2022

Continual Learning with Distributed Optimization: Does CoCoA Forget?

We focus on the continual learning problem where the tasks arrive sequen...
research
06/12/2019

Probing Multilingual Sentence Representations With X-Probe

This paper extends the task of probing sentence representations for ling...
research
04/22/2022

Memory Bounds for Continual Learning

Continual learning, or lifelong learning, is a formidable current challe...
research
06/04/2023

Towards Robust Feature Learning with t-vFM Similarity for Continual Learning

Continual learning has been developed using standard supervised contrast...
research
08/10/2018

Unsupervised Learning of Sentence Representations Using Sequence Consistency

Computing universal distributed representations of sentences is a fundam...
research
10/11/2022

Towards Structure-aware Paraphrase Identification with Phrase Alignment Using Sentence Encoders

Previous works have demonstrated the effectiveness of utilising pre-trai...
research
08/15/2023

A User-Centered Evaluation of Spanish Text Simplification

We present an evaluation of text simplification (TS) in Spanish for a pr...

Please sign up or login with your details

Forgot password? Click here to reset