Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Need

03/13/2023
by   Da-Wei Zhou, et al.
0

Class-incremental learning (CIL) aims to adapt to emerging new classes without forgetting old ones. Traditional CIL models are trained from scratch to continually acquire knowledge as data evolves. Recently, pre-training has achieved substantial progress, making vast pre-trained models (PTMs) accessible for CIL. Contrary to traditional methods, PTMs possess generalizable embeddings, which can be easily transferred. In this work, we revisit CIL with PTMs and argue that the core factors in CIL are adaptivity for model updating and generalizability for knowledge transferring. 1) We first reveal that frozen PTM can already provide generalizable embeddings for CIL. Surprisingly, a simple baseline (SimpleCIL) which continually sets the classifiers of PTM to prototype features can beat state-of-the-art even without training on the downstream task. 2) Due to the distribution gap between pre-trained and downstream datasets, PTM can be further cultivated with adaptivity via model adapting. We propose ADapt And Merge (ADAM), which aggregates the embeddings of PTM and adapted models for classifier construction. ADAM is a general framework that can be orthogonally combined with any parameter-efficient tuning method, which holds the advantages of PTM's generalizability and adapted model's adaptivity. 3) Additionally, we find previous benchmarks are unsuitable in the era of PTM due to data overlapping and propose four new benchmarks for assessment, namely ImageNet-A, ObjectNet, OmniBenchmark, and VTAB. Extensive experiments validate the effectiveness of ADAM with a unified and concise framework.

READ FULL TEXT

page 8

page 15

page 17

page 18

page 20

page 22

page 25

page 35

research
10/27/2022

Do Pre-trained Models Benefit Equally in Continual Learning?

Existing work on continual learning (CL) is primarily devoted to develop...
research
03/09/2023

Rethinking Visual Prompt Learning as Masked Visual Token Modeling

Prompt learning has achieved great success in efficiently exploiting lar...
research
03/28/2023

Large-scale Pre-trained Models are Surprisingly Strong in Incremental Novel Class Discovery

Discovering novel concepts from unlabelled data and in a continuous mann...
research
01/05/2021

One vs Previous and Similar Classes Learning – A Comparative Study

When dealing with multi-class classification problems, it is common prac...
research
03/17/2023

A Unified Continual Learning Framework with General Parameter-Efficient Tuning

The "pre-training → downstream adaptation" presents both new opportuniti...
research
03/09/2020

KGvec2go – Knowledge Graph Embeddings as a Service

In this paper, we present KGvec2go, a Web API for accessing and consumin...
research
11/23/2018

Revisiting Pre-training: An Efficient Training Method for Image Classification

The training method of repetitively feeding all samples into a pre-defin...

Please sign up or login with your details

Forgot password? Click here to reset