DeepAI AI Chat
Log In Sign Up

Theoretical Understanding of the Information Flow on Continual Learning Performance

by   Josh Andle, et al.

Continual learning (CL) is a setting in which an agent has to learn from an incoming stream of data sequentially. CL performance evaluates the model's ability to continually learn and solve new problems with incremental available information over time while retaining previous knowledge. Despite the numerous previous solutions to bypass the catastrophic forgetting (CF) of previously seen tasks during the learning process, most of them still suffer significant forgetting, expensive memory cost, or lack of theoretical understanding of neural networks' conduct while learning new tasks. While the issue that CL performance degrades under different training regimes has been extensively studied empirically, insufficient attention has been paid from a theoretical angle. In this paper, we establish a probabilistic framework to analyze information flow through layers in networks for task sequences and its impact on learning performance. Our objective is to optimize the information preservation between layers while learning new tasks to manage task-specific knowledge passing throughout the layers while maintaining model performance on previous tasks. In particular, we study CL performance's relationship with information flow in the network to answer the question "How can knowledge of information flow between layers be used to alleviate CF?". Our analysis provides novel insights of information adaptation within the layers during the incremental task learning process. Through our experiments, we provide empirical evidence and practically highlight the performance improvement across multiple tasks.


page 1

page 2

page 3

page 4


Continual Learning in Neural Networks

Artificial neural networks have exceeded human-level performance in acco...

Continual Learning via Bit-Level Information Preserving

Continual learning tackles the setting of learning different tasks seque...

Continual Classification Learning Using Generative Models

Continual learning is the ability to sequentially learn over time by acc...

A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrix

Continual learning (CL) is a setting in which an agent has to learn from...

Online continual learning with no task boundaries

Continual learning is the ability of an agent to learn online with a non...

Progress & Compress: A scalable framework for continual learning

We introduce a conceptually simple and scalable framework for continual ...

Learn to Bind and Grow Neural Structures

Task-incremental learning involves the challenging problem of learning n...