DRILL: Dynamic Representations for Imbalanced Lifelong Learning

by   Kyra Ahrens, et al.

Continual or lifelong learning has been a long-standing challenge in machine learning to date, especially in natural language processing (NLP). Although state-of-the-art language models such as BERT have ushered in a new era in this field due to their outstanding performance in multitask learning scenarios, they suffer from forgetting when being exposed to a continuous stream of data with shifting data distributions. In this paper, we introduce DRILL, a novel continual learning architecture for open-domain text classification. DRILL leverages a biologically inspired self-organizing neural architecture to selectively gate latent language representations from BERT in a task-incremental manner. We demonstrate in our experiments that DRILL outperforms current methods in a realistic scenario of imbalanced, non-stationary data without prior knowledge about task boundaries. To the best of our knowledge, DRILL is the first of its kind to use a self-organizing neural architecture for open-domain lifelong learning in NLP.



There are no comments yet.


page 1

page 2

page 3

page 4


Continual Domain Adaptation for Machine Reading Comprehension

Machine reading comprehension (MRC) has become a core component in a var...

Online continual learning with no task boundaries

Continual learning is the ability of an agent to learn online with a non...

Continual Lifelong Learning in Natural Language Processing: A Survey

Continual learning (CL) aims to enable information systems to learn from...

Unsupervised Continual Learning and Self-Taught Associative Memory Hierarchies

We first pose the Unsupervised Continual Learning (UCL) problem: learnin...

Continual Unsupervised Representation Learning

Continual learning aims to improve the ability of modern learning system...

Fine-tuning BERT-based models for Plant Health Bulletin Classification

In the era of digitization, different actors in agriculture produce nume...

Pitfalls of Static Language Modelling

Our world is open-ended, non-stationary and constantly evolving; thus wh...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.