Technical Report for ICCV 2021 Challenge SSLAD-Track3B: Transformers Are Better Continual Learners

01/13/2022
by   Duo Li, et al.
0

In the SSLAD-Track 3B challenge on continual learning, we propose the method of COntinual Learning with Transformer (COLT). We find that transformers suffer less from catastrophic forgetting compared to convolutional neural network. The major principle of our method is to equip the transformer based feature extractor with old knowledge distillation and head expanding strategies to compete catastrophic forgetting. In this report, we first introduce the overall framework of continual learning for object detection. Then, we analyse the key elements' effect on withstanding catastrophic forgetting in our solution. Our method achieves 70.78 mAP on the SSLAD-Track 3B challenge test set.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/10/2021

Elastic Weight Consolidation (EWC): Nuts and Bolts

In this report, we present a theoretical support of the continual learni...
research
04/17/2022

Continual Hippocampus Segmentation with Transformers

In clinical settings, where acquisition conditions and patient populatio...
research
05/08/2023

BiRT: Bio-inspired Replay in Vision Transformers for Continual Learning

The ability of deep neural networks to continually learn and adapt to a ...
research
11/15/2022

Exploring the Joint Use of Rehearsal and Knowledge Distillation in Continual Learning for Spoken Language Understanding

Continual learning refers to a dynamical framework in which a model or a...
research
07/23/2020

ADER: Adaptively Distilled Exemplar Replay Towards Continual Learning for Session-based Recommendation

Session-based recommendation has received growing attention recently due...
research
08/25/2023

ConSlide: Asynchronous Hierarchical Interaction Transformer with Breakup-Reorganize Rehearsal for Continual Whole Slide Image Analysis

Whole slide image (WSI) analysis has become increasingly important in th...
research
03/10/2023

Dynamic Y-KD: A Hybrid Approach to Continual Instance Segmentation

Despite the success of deep learning methods on instance segmentation, t...

Please sign up or login with your details

Forgot password? Click here to reset