CLEVA-Compass: A Continual Learning EValuation Assessment Compass to Promote Research Transparency and Comparability

10/07/2021
by   Martin Mundt, et al.
22

What is the state of the art in continual machine learning? Although a natural question for predominant static benchmarks, the notion to train systems in a lifelong manner entails a plethora of additional challenges with respect to set-up and evaluation. The latter have recently sparked a growing amount of critiques on prominent algorithm-centric perspectives and evaluation protocols being too narrow, resulting in several attempts at constructing guidelines in favor of specific desiderata or arguing against the validity of prevalent assumptions. In this work, we depart from this mindset and argue that the goal of a precise formulation of desiderata is an ill-posed one, as diverse applications may always warrant distinct scenarios. Instead, we introduce the Continual Learning EValuation Assessment Compass, CLEVA-Compass for short. The compass provides the visual means to both identify how approaches are practically reported and how works can simultaneously be contextualized in the broader literature landscape. In addition to promoting compact specification in the spirit of recent replication trends, the CLEVA-Compass thus provides an intuitive chart to understand the priorities of individual systems, where they resemble each other, and what elements are missing towards a fair comparison.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 3

09/14/2020

CVPR 2020 Continual Learning in Computer Vision Competition: Approaches, Results, Current Challenges and Future Directions

In the last few years, we have witnessed a renewed and fast-growing inte...
04/15/2021

Rehearsal revealed: The limits and merits of revisiting samples in continual learning

Learning from non-stationary data streams and overcoming catastrophic fo...
04/15/2019

Three scenarios for continual learning

Standard artificial neural networks suffer from the well-known issue of ...
10/30/2018

Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines

Continual learning has received a great deal of attention recently with ...
06/04/2021

A Procedural World Generation Framework for Systematic Evaluation of Continual Learning

Several families of continual learning techniques have been proposed to ...
10/09/2020

Continual learning using hash-routed convolutional neural networks

Continual learning could shift the machine learning paradigm from data c...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.