Real-Time Evaluation in Online Continual Learning: A New Paradigm

02/02/2023
by   Yasir Ghunaim, et al.
5

Current evaluations of Continual Learning (CL) methods typically assume that there is no constraint on training time and computation. This is an unrealistic assumption for any real-world setting, which motivates us to propose: a practical real-time evaluation of continual learning, in which the stream does not wait for the model to complete training before revealing the next data for predictions. To do this, we evaluate current CL methods with respect to their computational costs. We hypothesize that under this new evaluation paradigm, computationally demanding CL approaches may perform poorly on streams with a varying distribution. We conduct extensive experiments on CLOC, a large-scale dataset containing 39 million time-stamped images with geolocation labels. We show that a simple baseline outperforms state-of-the-art CL methods under this evaluation, questioning the applicability of existing methods in realistic settings. In addition, we explore various CL components commonly used in the literature, including memory sampling strategies and regularization approaches. We find that all considered methods fail to be competitive against our simple baseline. This surprisingly suggests that the majority of existing CL literature is tailored to a specific class of streams that is not practical. We hope that the evaluation we provide will be the first step towards a paradigm shift to consider the computational cost in the development of online continual learning methods.

READ FULL TEXT

page 2

page 4

research
03/20/2023

Computationally Budgeted Continual Learning: What Does Matter?

Continual Learning (CL) aims to sequentially train models on streams of ...
research
04/15/2020

Dark Experience for General Continual Learning: a Strong, Simple Baseline

Neural networks struggle to learn continuously, as they forget the old k...
research
03/29/2022

Online Continual Learning on a Contaminated Data Stream with Blurry Task Boundaries

Learning under a continuously changing data distribution with incorrect ...
research
07/11/2023

EgoAdapt: A multi-stream evaluation study of adaptation to real-world egocentric user video

In egocentric action recognition a single population model is typically ...
research
02/27/2019

Continual Learning with Tiny Episodic Memories

Learning with less supervision is a major challenge in artificial intell...
research
05/11/2022

A Continual Deepfake Detection Benchmark: Dataset, Methods, and Essentials

There have been emerging a number of benchmarks and techniques for the d...
research
10/30/2018

Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines

Continual learning has received a great deal of attention recently with ...

Please sign up or login with your details

Forgot password? Click here to reset