DeepAI AI Chat
Log In Sign Up

Scalable Recollections for Continual Lifelong Learning

by   Matthew Riemer, et al.

Given the recent success of Deep Learning applied to a variety of single tasks, it is natural to consider more human-realistic settings. Perhaps the most difficult of these settings is that of continual lifelong learning, where the model must learn online over a continuous stream of non-stationary data. A continual lifelong learning system must have three primary capabilities to succeed: it must learn and adapt over time, it must not forget what it has learned, and it must be efficient in both training time and memory. Recent techniques have focused their efforts largely on the first two capabilities while the third capability remains largely unexplored. In this paper, we consider the problem of efficient and effective storage of experiences over very large time-frames. In particular we consider the case where typical experiences are n bits and memories are limited to k bits for k << n. We present a novel scalable architecture and training algorithm in this challenging domain and provide an extensive evaluation of its performance. Our results show that we can achieve considerable gains on top of state-of-the-art methods such as GEM.


Continual Prototype Evolution: Learning Online from Non-Stationary Data Streams

As learning from non-stationary streams of data has been proven a challe...

Continual learning of longitudinal health records

Continual learning denotes machine learning methods which can adapt to n...

Continual Lifelong Learning in Natural Language Processing: A Survey

Continual learning (CL) aims to enable information systems to learn from...

Label-Efficient Online Continual Object Detection in Streaming Video

To thrive in evolving environments, humans are capable of continual acqu...

Real-Time Evaluation in Online Continual Learning: A New Paradigm

Current evaluations of Continual Learning (CL) methods typically assume ...

One Pass ImageNet

We present the One Pass ImageNet (OPIN) problem, which aims to study the...

Sequoia: A Software Framework to Unify Continual Learning Research

The field of Continual Learning (CL) seeks to develop algorithms that ac...