Energy-Based Models for Continual Learning

11/24/2020
by   Shuang Li, et al.
1

We motivate Energy-Based Models (EBMs) as a promising model class for continual learning problems. Instead of tackling continual learning via the use of external memory, growing models, or regularization, EBMs have a natural way to support a dynamically-growing number of tasks or classes that causes less interference with previously learned information. We find that EBMs outperform the baseline methods by a large margin on several continual learning benchmarks. We also show that EBMs are adaptable to a more general continual learning setting where the data distribution changes without the notion of explicitly delineated tasks. These observations point towards EBMs as a class of models naturally inclined towards the continual learning regime.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 8

page 15

04/04/2021

Understanding Continual Learning Settings with Data Distribution Drift Analysis

Classical machine learning algorithms often assume that the data are dra...
11/15/2021

Target Layer Regularization for Continual Learning Using Cramer-Wold Generator

We propose an effective regularization strategy (CW-TaLaR) for solving c...
04/22/2022

Memory Bounds for Continual Learning

Continual learning, or lifelong learning, is a formidable current challe...
10/22/2020

Continual Learning in Low-rank Orthogonal Subspaces

In continual learning (CL), a learner is faced with a sequence of tasks,...
05/27/2021

Encoders and Ensembles for Task-Free Continual Learning

We present an architecture that is effective for continual learning in a...
04/19/2021

Continual Learning with Fully Probabilistic Models

We present an approach for continual learning (CL) that is based on full...
06/24/2021

Continual Novelty Detection

Novelty Detection methods identify samples that are not representative o...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.